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ABSTRACT 

This handbook is designed to provide colleges and 
universities with initial guidance in establishing an appropriate 
system of indicators of undergraduate instruction, and to build on 
this foundation by cataloging a range of exemplary indicators of 
’’good practice” that have proven useful across many collegiate 
settings. It provides a brief overview of the rising demand for 
performance indicators at all levels of higher education and 
discusses the properties of a good indicator, emphasizing the ways 
that such measures can be used and abused* The handbook provides a 
taxonomy of ’’good practice” domains according to which useful 
indicators can be developed, and discusses various data sources that 
institutions can use to construct indicators of the kinds described 
in the handbook. The bulk of the handbook consists of 65 suggested 
’’good practice” indicators, each of which includes a brief 
definition, an explanation of its potential use and purpose, and the 
relevant data collection or calculational procedures required to 
produce it. (Contains 40 references.) (MDM) 



* Reproductions supplied by EDRS are the best that can be made * 

* from the original document . * 

*********************************************************************** 



ED 403 828 



s 



INDICATORS OF “GOOD PRACTICE” 
IN UNDERGRADUATE EDUCATION 





U.S. DEPARTMENT OF EDUCATION 

Offtce oi Educational Research and Improvomeht 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

his document has been reproduced es 
received (rom the person or oroanization 
Originating it. 

□ Minor changes have been made to improve 
reproduction quality. 

• Pointsof view or opinions stated m this docu- 
ment do not necessarily represent oHicial 
OERI position or policy 



A HANDBOOK FOR DEVELOPMENT 



AND IMPLEMENTATION 



"FERMISSiON TO REPRODUCE THIS ! 
MATERIAL HAS BEEN GRANTED BY I 



BEST COPY AVAILABLE 

^ ^ ^ NCHEMS 



2 



. .X 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



Indicators of “Good Practice” 
In Undergraduate Education: 
A Handbook for Development 
and Implementation 



Q 



INDICATORS OF “GOOD PRACTICE” 
IN UNDERGRADUATE EDUCATION: 

A HANDBOOK FOR DEVELOPMENT 
AND IMPLEMENTATION 



Peter T. Ewell 
Dennis P. Jones 



1996 




National Center for Higher Education Management Systems (NCHEMS) 
P. O. Box 9752 Boulder, Colorado 80301-9752 



This handbook was not printed at 
the expense of the federal government. 
Printed in the United States of America. 



1996 

NATIONAL CENTER FOR HIGHER EDUCATION 
MANAGEMENT SYSTEMS 
Boulder, Colorado 8030 1 -9752 
Designed by Paula Dressier and Clara Roberts 




Table of Contents 

Chapter 1: Introduction and Overview 1 

Why Indicators? ^ 

Indicators and Information-Based Management 2 

Why Indicators of “Good Practice” 5 

Chapter 2: What Makes a Good Indicator? 9 

Some Attributes of Indicators 

Some Resulting Principles of Design 12 

Chapter 3: Indicators of What? The Domains of “Good Practice” 17 

Specific Domains of “Good Practice” 18 

Some Cross-Cutting Indicator Domains 23 

Indicators and Outcomes 24 

Chapter 4: Data Sources for “Good Practice” Indicators 29 

Types of Data Sources 29 

Developing a “Quality Management Database” 33 

Presenting Indicators Results 36 

Chapter 5: Some Concluding Thoughts 39 

A Catalog of “Good Practice” Indicators 41 

References 10^ 



O 

ERLC 



6 



Figure 1 - 
Figure 2 - 
Figure 3 - 
Figure 4 - 



Figures 



A Conceptual View of the Management Process 

A Taxonomy for Developing “Good Practice” Indicators 
Summary Chart of Potential “Good Practice” Indicators 
Types of Databases for Academic Management 



Foreword and Acknowledgments 



T he NCHEMS core mission is to provide institutions and agencies of higher education with proven 
informational tools for planning, management and evaluation. This Handbook is designed to further that 
mission by extending the coverage of management information to the actual delivery of instruction. Like other 
NCHEMS publications, its contents are based on practical experience gained through direct consulting with 
institutions and in a range of research and development projects. Particularly to be acknowledged in the first 
category is recent work with Winona and Northeast Missouri (now Truman) State Universities, where many 
of the suggested approaches and measures were piloted. Prominent in the latter category are projects on 
accountability and performance indicators undertaken for the Education Commission of the States (ECS), the 
Colorado Commission on Higher Education (CCHE), and the U.S. Department of Education’s Office of 
Institutional Research and Improvement (OERI). In addition, we want especially to thank C. Robert Pace 
whose excellent work in developing the College Student Experience Questionnaire originally inspired our 
efforts, and who gave us good advice on many occasions. 

Because the Handbook is part of an ongoing developmental effort on instructional indicators, NCHEMS 
welcomes any additions to the Catalog that institutions may have tried, reports of experience in trying out such 
indicators, and additional suggestions for improvement. Please feel free to write or to contact us directly 
through our Web Page at http://www.nchems.com. 



Chapter 1 



Introduction and Overview 



U nder the changing internal and external 
conditions faced by most of those charged 
with the academic management of colleges and 
universities, a principal imperative is simply to 
know what is going on. Explosions in academic 
knowledge, together with growing pressure to 
respond to the multiplying demands of an expanding 
array of clients, have caused growing diversity in 
both curricula and instructional methods. The 
resulting size and complexity of today’s “instruc- 
tional delivery system” makes it far less easy to 
manage by touch and feel. At the same time, the 
need to stretch increasingly scarce resources across 
this expanding territory has led to equally pressing 
demands to do more with less. Here as well, the 
managerial need is for more and better information 
about how things are working, and how they 
might be made to work better. Information about 
instructional delivery is also fast becoming a 
centerpiece of public accountability. Pressed by 
both students and taxpayers, public officials are 
insisting that institutions report information that 
reflects what they are actually doing in under- 
graduate education and what students will get as a 



result, in addition to more traditional measures of 
instructional resources and investments. 

Given these forces, growing interest in the topic 
of performance indicators as both tools for academic 
management and as key ingredients of a new 
accountability should come as no surprise. The 
purpose of this Handbook is to provide colleges and 
universities with initial guidance in establishing an 
appropriate system of indicators of undergraduate 
instruction, and to build on this foundation by 
cataloging a range of exemplary indicators of “good 
practice” that have proven useful across many 
collegiate settings. Chapter 1 provides a brief 
overview of the rising demand for performance 
indicators at all levels of higher education, and 
discusses the underlying management philosophy 
within which any set of institution-level indicators 
of academic practice ought to be developed. In the 
light of campus experience with assessment, the 
chapter also makes the specific case for developing 
indicators of undergraduate “good practice” in 
preference to more traditional resource-based and 
descriptive indicators of instructional delivery and 
as a supplement to information about effectiveness. 
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Chapter 2 discusses explicitly the properties of a 
good indicator, and emphasizes the ways such 
measures can be used and abused. Chapter 3 
provides a taxonomy of “good practice” domains 
according to which useful indicators can in fact be 
developed, based on factors known through the 
research literature to be associated with collegiate 
achievement. Finally, Chapter 4 discusses various 
data sources that institutions can use to construct 
indicators of the kinds described in the Handbook, 
and provides suggestions about how to present such 
information and organize and archive the resulting 
database. 

With these chapters as context, the core of the 
Handbook consists of a catalog of suggested “good 
practice” indicators. Organized in terms of the 
taxonomy suggested in Chapter 2, institutions are 
encouraged to adopt or adapt these sample measures 
in developing their own approaches. Each entry in 
the catalog describes the proposed measure by 
providing a brief definition, an explanation of its 
potential use and purpose, and the relevant 
data-collection and/or calculational procedures 
required to produce it. Each indicator so defined in 
the catalog has been piloted in the NCHEMS work 
with individual institutions, though no formal claims 
of validity or reliability can at this point be made. 
As a result, the primary intent of the catalog is to 
stimulate good thinking at the institutional level, and 
to provide a place for each institution to start in 
developing its own indicators approaches. 

Why Indicators? 

Escalating interest in performance indicators in 
higher education stems from a number of sources. 
A first stimulus is the fact that state and federal 
authorities — as well as institutional accrediting 
bodies — ^have in the last decade become far more 
specific in their demands for information. In the 
early to mid-eighties, such demands centered largely 
on the unexplored and controversial territory of 
academic outcomes (Ewell 1991). More recently, 
such demands have become more focused — 
centered prominently on the need to allocate scarce 
public resources and to inform individual “consumer 
choices” about which college or university to attend 



and about the specific experiences associated with 
attendance (Ewell 1994, ECS 1995). Two things 
have happened as a result. First, the external market 
for performance information has gradually ex- 
panded from an exclusive concern with outcomes to 
encompass questions of instructional practices as 
well. Second, the particular form in which in- 
formation of all types has been requested by 
external actors has become increasingly terse and 
truncated — placing a premium on quantitative, 
easily-compared measures of condition or per- 
formance (Ruppert 1994). Taken together, these 
two phenomena have yielded a growing number of 
statistical “report-cards” for higher education, 
prepared by sources ranging from state governments 
to US News and World Report. 

A second stimulus for the use of such indicators is 
internal. Faced with growing fiscal pressures, but 
unabated student demand, many colleges and 
universities are beginning to restructure and adopt 
management approaches based on the principles of 
Total Quality Management (TQM). Initially con- 
fined to administrative functions, TQM concepts 
(though with many caveats) are gradually finding 
their way into the academic arena (Seymour 1991, 
Sherr and Teeter 1991). In contrast to assessment’s 
apparent emphasis on “inspection at the end,” such 
approaches emphasize continuously gathering data 
about how “core processes” are actually working, 
and focus especially on the ways in which appar- 
ently different functions are supposed to fit together 
to achieve common ends (Ewell 1993). This 
emphasis is quite consistent with curricular reform 
themes of the last decade that stress coherence and 
integration among planned sequences of courses. 
But it also demands concrete information in order to 
actively monitor such connections and to determine 
where appropriate improvements can be made. 

Indicators and 

Information-Based Management 

Performance indicators must also be seen as part 
of a wider management philosophy — one that 
embraces goals, inputs, institutional activities, and 
outputs. Figure 1 emphasizes this relationship by 
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noting the specific manner in which each of these 
components is linked. Each linkage requires 
managers to possess sound information if the 
process is to be effective. As emphasized by the 
figure, overall effectiveness at any level of analysis 
within the institution depends upon an array of 
factors, and it is in many ways the alignment 
among them that must be monitored. If intended 
instructional goals are to be achieved, resources 
must be committed appropriately in the form of a 
budget, be appropriately allocated, and result in 
outcomes consistent with original intent. If any part 
of this evolving chain of management breaks, the 
achievement of intended results is unlikely — and 
simply knowing the ultimate result will tell the 
manager little about the particular stage in the 
process at which the failure occurred. An important 
consequence is that adequate indicator systems must 
reflect all four aspects of this generic management 
cycle. 

The various informational components of the 
overall scheme represented by Figure 1, however, 
have developed unevenly. In fact, their evolution 
reflects both the capabilities of evolving data- 
collection technology and the issues that dominated 
higher education decisionmaking at various points 
in time. Thus the seventies primarily emphasized 
the development of costing and resource-allocation 
models and the design of expansion-oriented long- 
range planning approaches. Greater resource 
constraints in the late seventies and early eighties led 
to the additional development of program review 
systems designed to emphasize efficiencies, and to 
early concerns with educational outcomes and the 
assessment of institutional effectiveness. This 
movement, in turn, was more fully developed in the 
late eighties — a period concerned especially with 
the improvement of higher education’s quality, 
especially in the realm of undergraduate education. 
All these topics are combined in today’s man- 
agement environment where “quality” remains 
expected by all, but at reduced cost and defmed as 
much by higher education’s “customers” as by 
colleges and universities themselves. 

Across all these periods, however, a number of 
key concepts and analytical distinctions about 
linking information to decisions have endured. And 



these remain as valid in developing indicators of 
institutional good practice and performance as when 
they were first put forward some thirty years ago. 
Among these informational “first principles” are the 
following: 

• the distinction between data and information. 
Though simple, this distinction is fundamental to 
any approach to management information (Jones 
1982). One basic insight that underlies it is the 
consistent need to relate pieces of data — often 
drawn from quite different sources — in order to 
say anything meaningful about institutional 
condition or performance. Asa result, most useful 
indicators are based on ratios or similar cal- 
culations which may be not only unfamiliar to 
many constituencies but which may also be 
technically difficult to construct. 

• cost/benefit “Performance” at any level requires 
examining inputs, activities, and outcomes 
together. It also requires explicit consideration of 
the multiple perspectives on each of these held 
by different constituents and stakeholders — 
especially who benefits and who pays. From an 
evaluative standpoint, this means carefully 
examining ratios of resources expended to par- 
ticular units of activities undertaken and benefits 
received — and it is important to remember that 
many of the entities on both sides of this ratio may 
be non-material or at least non-monetary. From a 
planning perspective, moreover, it is important to 
provide clear linkages between information about 
performance and further resource investments. 
An important insight in both contexts is that 
useful indicators will virtually always be 
perspective-driven, and demand attention to 
the differing informational needs of different 
audiences. 

« a focus on functions and results as well as 
organizational units. “Performance” and “good 
practice” at any level also tend to defy class- 
ification by organizational unit or traditional 
line-item budgetary categories. As a result, it is 
important to avoid adopting approaches to man- 
agement information that are rooted exclusively 
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Figure 1 



A Conceptual View of the Management Process* 




* Expanded version of a framework developed by Leonard C. Romney, Robert F. Gray, and H. Kent Weldon. "Departmental 
Prodnctivity: A Conceptnal Framework." (Unpnblished paper). Bonlder, Colo.: NCHEMS, May 1978. 
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in existing organizational categories. The most 
useful approaches to system development instead 
center on the notion of defining goals (or inten- 
ded good effects), then determining the degree 
to which defined objectives have in fact been 
obtained through the deployment of identified 
assets or as a consequence of undertaking 
particular activities. Units of analysis for 
describing good practices or for assessing 
performance are thus bounded by such categories 
as “objectives,” “functions,” and “processes” 
more than they are by particular actors or 
organizational units. For the construction of 
indicators, a primary implication that will be 
stressed throughout this Handbook is their use in 
combination — to present an overall picture of 
how things are functioning in different parts of 
an institution’s overall system of instructional 
delivery. 

• choosing the proper unit of analysis. A major 
conceptual trap embedded in the development of 
statistical indicators is that of suboptimization. 
The same piece of data may have quite different 
meanings at different levels of analysis, depending 
upon the configuration or function of the overall 
system. In the development of appropriate 
indicators of instructional delivery, this may be 
most strongly apparent in the need to distinguish 
among the appropriate performance domains of 
individual academic departments on the one hand, 
and those of such institution-wide functions as 
advising and general education on the other. The 
latter are not simply further aggregations of the 
former, but may require the specification of 
domain elements of quite different kinds. The 
same levels of “performance” on similar dim- 
ensions also may not mean the same things at 
different levels. From a systems perspective, for 
instance, institutional interests may not be well 
served by a set of departments that all define and 
maximize “quality instruction” entirely in self- 
defined ways. 

• emphasizing the linkage between information 
and its particular uses and users. Information 
about “good practice” or “performance,” more- 



over, makes little sense in the abstract. To be of 
value, it must be embedded directly in a visible 
management or decisionmaking process. This 
axiom seems close to simplistic, but it has 
profound implications for the properties of good 
information. One such implication is that the 
precision of the information required for manage- 
ment purposes is dictated principally by the para- 
meters (and the risks associated with) a particular 
application or decision to be made. It is not 
therefore a strictly “methodological” question. A 
second implication is that any resulting indicator 
must have “face validity.” That is, it must be 
perceived by the user as a relevant and appropriate 
measure of the phenomenon at issue, regardless of 
its technical properties. Finally, to be useful as an 
indicator, any piece of information should sug- 
gest what needs to be done in order to improve 
performance. 

These themes have been present in most sound 
discussions of management information in higher 
education settings since the mid-sixties. For those 
attempting to develop good indicators they should 
remain living principles, applied flexibly and used 
in combination to meet particular management and 
measurement contexts. 

Why Indicators of “Good Practice?” 

Since about 1985, most discussions about 
information-based accountability and reform in 
undergraduate education have centered on the 
assessment of outcomes. While this focus has been 
both timely and appropriate given the paucity of 
such information, it has also encountered many 
drawbacks. Because of its visible linkage to 
accountability, of course, institutional resistance to 
assessment has been common. At the same time, the 
technical difficulties of establishing adequate and 
credible assessment systems are formidable, and 
frequently result in assigning assessment to 
technical specialists rather than those actively en- 
gaged in instruction and curricular development. 
Partly as a result, assessment at many institutions 
has all too often become a “train on its own track,” 
with little real connection to either academic 
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policymaking or the improvement of teaching and 
learning. 

Given this context, three particular difficulties 
that institutions have faced in implementing 
assessment are worth noting: 

* high costs and long timelines for developing 
appropriate assessment instruments. After an 
initial encounter with standardized testing, most 
institutions embark upon a multi-year process 
of exploring alternative “performance-based” 
approaches. While innovative and ultimately 
beneficial, they have also found developing, 
validating, and piloting such assessments to be 
expensive and time-consuming. The technical 
properties of performance-based assessments are 
complex and in many cases unknown: standard 
validity and reliability measures are hard to apply 
and the results are often subject to unknown and 
uncontrollable biases. Growing experience with 
assessing student populations in specially- 
constructed settings, moreover, suggests that 
motivating students to participate and do their best 
may be a major problem (Banta and Associates 
1993). These challenges are gradually being 
overcome, but for all too many colleges and 
universities the promised payoffs of outcomes 
assessment seem far down the road. Meanwhile, 
alternative sources of information about per- 
formance may be needed. 

* the need to gain consensus about instructional 
goals. One of the most formidable challenges 
involved in assessment is forging real agreement 
about intended outcomes. Abilities like “critical 
thinking” or “problem-solving” prominently 
proclaimed in most institutional goal statements, 
are operationally ill-specified and can mean 
profoundly different things to academic and 
non-academic audiences (ECS 1995), As a result, 
such attributes can be operationalized in terms of 
quite different kinds of assessment approaches. 
As many have noted (e.g., Banta and Schneider 
1988), the process of goal-development forced by 
the need to engage in assessment can itself be 
beneficial, and it is well worth the investment 
involved to bring it to a conclusion. But the 



process takes much time and commitment. In 
addition, it encounters an ongoing tendency to 
continually re-open discussions about the validity 
of the particular instruments chosen to opera- 
tionalize such complex abilities, whatever they 
may be. 

• frequent inability to direct improvement. 
Probably the most important drawback to 
using the results of outcomes assessments for 
improvement — despite their often sound con- 
struction — has been the difficulty of linking them 
to prior educational experiences that can be 
changed or improved by management action. 
Valid test scores can indicate rather precisely 
what has been accomplished and where specific 
deficiencies exist, but they often provide fixis- 
tratingly little guidance about what can and should 
be done. And this limitation has proven par- 
ticularly applicable to the kinds of broad-based, 
higher-order abilities frequently claimed as 
outcomes of undergraduate education. 

These drawbacks have led many observers 
to recommend the development of additional 
data-gathering efforts centered on instructional 
practices and student experiences as an integral part 
of the assessment process (Pace 1984, Ewell 1996, 
Astin 1991). This case rests on two main grounds. 
First, such measures may provide important addi- 
tional information which can help both academic 
managers and policymakers make sense of the 
findings of end-point assessments. If their general 
validity can be established, such indicators can not 
only “supplement” information derived fi-om direct 
performance measures, but can be useful in then- 
own right. Except as a pure benchmark of progress, 
it makes little managerial sense to collect outcomes 
information in the absence of parallel information on 
key processes that are presumed to contribute to 
the result. Colleges and universities, moreover, are 
especially in need of good information about 
educational contexts and practices because of then- 
enormous variety. Not only do institutions of higher 
education exist in many forms with diverse 
educational missions, but unlike the situation typi- 
cal of K-12 education, there is little curricular 
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commonality in the experiences of college students. 
Given this situation, information on outcomes 
alone is virtually uninterpretable in the absence 
of information about key experiences. 

Second, a set of indicators tied to important 
institutional practices provides clear guidance for 
action. As noted, a major difficulty experienced by 
institutions throughout the brief history of 
assessment has been the relative lack of direction 
that its results provide about exactly where to invest 
resources in order to obtain desired results. More 
effective in causing meaningful change have been 
simultaneous approaches that focus on what 
institutions and faculty actually do. Prominent 
examples in recent years have been the “Seven 
Principles for Good Practice in Undergraduate 
Education” (Chickering and Gamson 1987) and 
“classroom assessment” techniques (Angelo and 
Cross 1993) — both of which have been widely 
adopted across many settings without substantial 
faculty resistance. Measures based on approaches 
such as these are likely to be of considerable value 



in determining the degree to which institutions and 
academic units are willing and able to act con- 
sistently in support of undergraduate educational 
practices that are both agreed upon by the wider 
academic community, and are known to work. 

Evolving experience with using such indicators 
suggests that this original case is justified. 
Appropriate indicators of “good practice” that can 
be empirically related to desired collegiate outcomes 
and that can provide institutions with concrete 
directions for improvement are feasible and the 
technology needed to implement them is available 
(NCHEMS 1994). Their increasing use in academic 
settings testifies to their growing utility to academic 
managers when used for these purposes. The 
chapters that follow are designed specifically to 
help academic managers understand, design, and 
implement such indicators — especially in the realm 
of instructional delivery. And the accompanying 
catalog of potential indicators should in turn provide 
them with a concrete place to start. 
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Chapter 2 



What Makes A Good Indicator? 



T he term “indicator” has been used for many 
years to describe a relevant, easily-calculable 
statistic that reflects the overall condition of an 
enterprise or the progress of a particular set of events 
(Burstein, Oakes and Guiton 1992). Among the 
most prominent national examples are the U.S. 
Department of Labor’s unemployment rate and the 
U.S. Department of Commerce’s report on Gross 
Domestic Product (GDP), both of which are used to 
assess the nation’s overall economic health. Other 
familiar examples are in the field of health care. For 
example, the infant mortality rate is used in- 
ternationally as a rough gauge of the effectiveness 
of the overall public health of a nation or culture. 
Indicators such as these have been developed for a 
variety of reasons, most of them also applicable to 
higher education. And many of the lessons learned 
about their use and abuse are equally applicable. 

More recently, statistical performance indicators 
have been used in industry and health care to monitor 
and improve processes of manufacture and service 
delivery, largely because they can provide a reliable 
base for tracking and improving quality (Gitlow and 
Gitlow 1 987). Examples here are varied, but include 



such measures as component failure rates for 
manufactured products, or cycle times for providing 
particular services or responding to customer 
requests. Many of these latter techniques are already 
beginning to be applied to such eireas as student 
registration or business operations in college and 
university settings (Seymour 1991; Sherr and Teeter 
1991). 

To be useful as an indicator, a particular piece of 
data must communicate something important about 
what is happening in a complex domain. But to 
achieve this end it is not necessary for the statistic to 
be causally related to what it is intended to reflect. 
As noted, the infant mortality rate is commonly used 
as an indicator of the overall health of a nation or 
culture but it would not be appropriate to concentrate 
all resources on delivering prenatal health care in an 
attempt to improve this overall condition. Because 
of this, indicator statistics are often labelled 
“proxies” or “indirect measures” of condition or 
attainment, and caution in their use is advised. For 
similar reasons, indicators are most effective when 
used in combination. Indeed, the best of them 
are carefully constructed to provide mutually- 
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reinforcing measures that only together provide an 
accurate picture of what is occurring (Ewell and 
Jones 1994). 

Some Attributes of Indicators 

Like information in any form, such measures are 
good for some things and inappropriate for others. 
In general, experience suggests that indicators of 
instructional delivery are particularly useful for 
three main purposes: 

• to quickly compare relative performances across 
units, institutions or settings. 

• to monitor what is happening within a particular 
unit, institution or setting over time. 

• to explicitly examine the effects of intervention 
or policy change — either across settings or over 
time. 

Using indicators effectively also implies adopting 
a particular model of how effective management 
occurs. An indicators-based management approach 
first implies a conviction that it is legitimate for 
academic leaders to actively guide and restructure 
the academic enterprise in order to improve its 
performance. This conviction is only beginning to 
come into vogue in higher education and is strongly 
opposed to the traditional decentralized, often 
haphazard, administrative styles now typical of most 
college and university settings. Using indicators 
also implies adoption of a consciously incremental 
policy approach that stresses continuous im- 
provement over one-time “quick fix” or “magic 
bullet” solutions. This in turn implies a rational 
approach to policy and resource allocation, char- 
acterized by the explicit use of concrete information 
in deciding what can and should be done. An 
important implication of this approach is that 
considerable management attention be directed 
toward how an institution’s diverse instructional 
activities work together as a system — emphasizing, 
for instance, how their individual pieces interact, 
rather than using statistics more narrowly in an 
attempt to “micro-manage” the actions of individual 
units or subftmctions. 



These attributes also imply some rather specific 
strengths and weaknesses of indicators as man- 
agement tools — ^particularly in collegiate instruc- 
tional settings. Some important advantages of 
indicators are that they: 

* can help mobilize concerted action. Making 
goals concrete allows necessarily dispersed 
actions to be better aligned at all levels. Keeping 
track of known good practices such as the active 
learning experiences reported by students or the 
level of their out-of-class contact with faculty 
members, for instance, can send a much more 
explicit message to a collegiate community about 
what kinds of actions are expected of its members 
than simply encouraging them to be more “student 
centered.” 

* can help the institution communicate its goals 
explicitly to potential students and the public. 

Making intended outcomes and behaviors visible 
and concrete also allows the institution to better 
inform its “customers” about what it can deliver 
and what they can expect as a consequence. 
Providing information on the completion and 
occupational placement rates of former students, 
for instance, sends potential students a powerful 
message about effectiveness — especially if it is 
broken down by particular clienteles in a manner 
that allows them to make meaningful consumer 
choices about whether or not to attend. 

* can support and reinforce academic planning 
directed toward continuous improvement. 
Institution-wide indicators about instructional 
effectiveness and delivery can help document 
what actions are needed and can help chart 
progress in meeting identified needs. Com- 
prehensive data about the pass-rates in regular 
collegiate math courses for students completing 
remedial study for instance or the first-year 
dropout rates of minority students with identified 
incoming conditions can not only help to make the 
case for greater investment of resources to 
alleviate these conditions, but can also show 
explicitly the levels, kinds, and levels of utilization 
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of academic resources that will be needed to meet 
the remediation challenge across the entire 
institution. 

Academic leaders should also realize that in- 
dicator systems also have important drawbacks as 

management tools. Some particular pitfalls include: 

• their tendency to create false incentives for 
action. If the stakes associated with “poor 
performance” are high, faculty and unit-level 
administrators may act to maximize the numeric 
values of indicators without really changing 
what they do. Placing excessive emphasis on 
achievement test scores in examining learning 
outcomes, for example, can strongly induce 
faculty to “teach to the test” or may encourage 
individual academic programs to eliminate weak 
students before they are tested. 

• their tendency to focus attention on infor- 
mation-gathering itself, rather than action 
to change conditions. Because performance 
measures are often technically complex and 
difficult to implement, excessive institutional 
effort may be spent on measurement issues that are 
not related to improving instruction. Initiating 
complex centralized data-col lection systems to 
monitor student behaviors, for example, may 
require significant new institutional investments 
without eliminating the need for individual units 
and programs to maintain their own, often 
duplicative, assessment systems that are better 
suited to local requirements. 

• their frequent inability to tell outsiders what 
they really want to know. By their very 
nature, indicators tend to be indirect. Through a 
well-designed indicator system, for instance, 
institution-level administrators may be able to 
monitor which programs are investing heavily and 
appropriately in educational good practice, what 
kinds of resources they are investing, and what 
many outcomes may be. But they probably will 
not be able to develop a reliable single indicator of 
the overall “quality” of the education that the in- 
stitution provides that will respond to the diverse 
needs of many potential “customers.” 



Across all settings, moreover, important caveats 
are associated with the use of statistical indicators of 
any kind. First, indicators should not be used singly 
or in isolation. Instead, the most effective systems 
contain multiple measures that are designed from the 
outset to be mutually reinforcing. The information 
contained in a given indicator, moreover, almost 
always lies in comparison — of its value across 
different settings, at different points in time, or 
before and after intervention. Basing any conclusion 
on a single observation can be dangerous in the 
extreme. 

The specific settings across which indicators are 
compared must also be sufficiently similar to allow 
meaningful conclusions to be drawn. In makes little 
sense, for instance, to directly compare the student 
retention rates of a community college and a 
selective liberal arts college because of substantial 
differences in assigned mission and student 
clientele. At the very least, additional indicators 
should be constructed that reflect these differences, 
so that they can be taken directly into account in 
interpreting performance (in the example noted, for 
instance, an additional indicator of entering student 
ability might be constructed). 

Third, effective indicator systems should embrace 
multiple perspectives including those of the in- 
stitution (or academic unit), students, and such 
external constituents as employers, elected officials 
and taxpaying citizens. Appropriate policy and 
concerted action designed to meet “customer needs” 
demands consensus among these various peirties- 
at-interest about how goal achievement and effec- 
tive practice will be explicitly recognized when it 
occurs. Often the same basic information must 
be communicated in the form of different indicators 
to inform different audiences with varying decisions 
or judgments to make. For example, an overall 
degree-completion rate may serve as one indicator 
of institutional performance, but it provides poten- 
tial students with very little information about their 
own chances of graduating in the absence of far more 
specific information about the experiences of 
students like them. 

For use in improving instruction, indicators are 
at their best when they are used to raise questions 
or to identify potential problems; they are less well 



11 



suited for rendering summative judgements about 
adequacy or performance. The primary emphasis 
should therefore be placed not on “policing” unit- 
level or individual performance, but rather on 
determining the strengths and weaknesses of the 
institution’s system of instructional delivery as a 
whole. As a result, measures of “good practice” of 
the kinds described in this Handbook should always 
be accompanied by in-depth follow-up inves- 
tigations, in order to determine what in particular 
should be done about any problems identified. 

For use in informing external constituencies, in 
turn, indicators are at their best when used in 
combination to shed light on particular issues that 
may be uniquely associated with specific decisions 
or perspectives. The primary emphasis here should 
therefore be placed on determining more precisely 
what these decisions or perspectives may be and 
developing (and communicating) measures that 
respond directly to them. Prospective students, for 
instance, may prefer information focused especially 
on experiences typical of students like themselves, 
employers may be interested in specific curricular or 
pedagogical features related to their own work- 
places, and legislators and boards more focused on 
broader institutional investments and engagement in 
effective instructional practices. As a result, “good 
practice” measures of the types described should 
always include careful consideration of who the 
specific audience is intended to be. 

Like any policy tool, institutional indicator 
systems are no panacea. At the broadest level, their 
appropriate use demands careful planning and 
significant attention to how the resulting in- 
formation is aligned with institutional values and 
directions. More particularly, crafting good in- 
dicators requires considerable attention to the 
specific incentives and disincentives for action that 
the resulting system creates at all levels of the 
institutional community. 

Some Resulting Principles of Design 

Given these many cautions, academic ad- 
ministrators should consider a number of explicit 
questions in designing indicators. A potential 
measure may look good on one dimension but 
possess severe handicaps on another. So long as its 



specific strengths and weaknesses are known, 
however, a “flawed” statistic may still be useful for 
purposes of policy or planning. If a posed set of 
indicators fails to meet too many such criteria, 
however, the result may be substantial future 
difficulties. In developing effective indicators, 
academic leaders must fully be aware of their 
specific properties, and must resist the common 
temptation to view every such statistic as equally 
valid and useful for all purposes. 

The following design principles have proven 
appropriate in making such decisions across a wide 
range of settings. Taken together, they constitute a 
robust set of criteria against which any proposed set 
of indicators can appropriately be assessed: 

• leverage for action. This criterion addresses the 
degree to which a given indicator provides 
concrete guidance about what should be done. 
Some indicators merely provide information about 
a present state or condition without providing 
much guidance about how to improve things; 
others will point toward specific practices that 
might be changed. A comprehensive assessment 
of collegiate-level communications skills admin- 
istered to a sample of sophomores, for instance, 
may reliably indicate current levels of proficiency 
in written communication among students at this 
level across the institution. But it usually yields 
little information about what needs to be done to 
improve performance. In contrast, a statistic 
indicating the actual amounts of writing typically 
required in different academic majors and core 
courses, while it does not show proficiency 
directly, can provide faculty and department heads 
with much clearer direction about appropriate 
expectations. 

• vulnerability to manipulation. This principle 
concerns the extent to which the numeric value of 
the indicator in question can change (or can be 
made to change) without any real alteration in 
what it is designed to measure. All such measures 
are to some extent vulnerable to manipulation, and 
their robustness should be highly prized. Where 
the consequences of obtaining “low” values on 
such measures are perceived to be considerable, 
there is considerable pressure to maximize 
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indicator values in any way possible, ignoring the 
actual practices that the indicator is supposed to 
reflect. As a result, systems of indicators should 
be designed to reinforce one another. At the 
same time, single indicators should be avoided — 
especially if their individual values can be easily 
influenced. Statistics on the proportion of instruc- 
tional expenditures dedicated to delivering lower- 
division coursework, for instance, depend greatly 
on the ways individual courses are classified. If 
incentives to perform well on such indicators are 
compelling, departmental administrators will 
quickly identify those modes of classification that 
yield maximum indicator values and behave 
accordingly — regardless of what they may 
actually be doing. 

• credibility. This property reflects the degree to 
which the informational content of a given 
indicator is likely to be considered trustworthy by 
a particular audience or user. In part, this is related 
to the informational source or method of data 
collection used. External audiences such as 
policymakers and prospective students, for 
instance, will find information provided directly 
by former students and employers considerably 
more credible than institution-generated sta- 
tistics — especially if such information is collected 
by a “third party” agency. In part, however, this 
property is associated with the way particular 
measures are constructed. The best indicators for 
communicating with external audiences, for 
instance, are not only technically sound but “face 
valid” — that is, they not only measure what they 
are supposed to measure, but they look like they 
do and are thus far more likely to be believed than 
are equally valid statistics that lack this property. 

• ease of interpretation. This property reflects the 
degree to which the indicator conveys a clear and 
consistent meaning to its intended audiences — 
both within and outside the institution. Some 
indicators are the result of complex statistical 
calculations that raise many issues for those 
unschooled in their interpretation. Others are 
transparent to most observers and can com- 
municate readily to audiences drawn from a 
wide variety of backgrounds. Because their 



construction depends upon many prior assump- 
tions, statistics on program completion, for 
instance, are notoriously complicated when 
compared to more straightforward information 
about enrollments or numbers of faculty. Which 
students should in fact be included in the 
denominator of the calculation — those who 
entered the program as freshmen, those who 
transferred into or out of it from other majors, or 
some combination? At the same time, many 
statistics commonly used as indicators lack 
consensus about what constitutes “good” 
performance. Above-average class sizes in a given 
department or program, for instance, might 
suggest high efficiency (presumably a good thing) 
as well as low learning effectiveness (presumably 
a bad thing). In contrast, the directionality of such 
statistics as the average number of hours-per-week 
that students report devoting to academic matters 
is straightforward. In short, effective indicators 
must be transparent to their intended audiences, 
with their definitions and grounding assumptions 
communicated clearly and publicly each time the 
indicator is used. 

• balance of perspective. This criterion concerns 
the extent to which the indicator embodies the 
respective points of view of each of the meiny 
audiences to whom it is addressed. A good 
indicator system will contain information that is 
applicable to several levels of analysis (insti- 
tutional, departmental, etc.), and will consist of 
statistics designed to reflect a range of client 
perspectives including the student, the instruc- 
tional unit, and external audiences like taxpayers 
and public officials. A statistic on the employ- 
ment rates of program-completers by field of 
study, for instance, can be summarized from the 
student’s point of view as the probability of being 
placed in a rewarding job that in fact requires the 
skills he or she has learned, from the program’s 
point of view as an overall placement rate within 
the occupation for which the program is designed 
to provide training, and from an employer’s point 
of view as the proportion of needed jobs in 
particular skill areas that are filled each year. 
Because consensus is required for meaningful 
action, moreover, the contents of any indicator 
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system should stimulate active dialogue among 
the appropriate parties-at-interest in deter- 
mining academic policy. Failure rates in basic 
engineering or science courses that require a 
knowledge of calculus by level of prior math per- 
formance, for instance, can provide an excellent 
starting point for conducting meetings between 
math and engineering faculty aimed at improving 
the design and delivery of prerequisite courses. 

• available benchmarks or standards for com- 
parison. This principle addresses the particular 
metrics or standards that will be used to chart 
progress or success when using the statistic in 
question. The meaning of any reported statistic 
comes only when its value is assessed against 
something else. But appropriate comparisons may 
vary considerably depending upon intended 
purpose. An indicator like the deployment of 
faculty time, for instance, can be first compared 
across all departments to determine (with 
appropriate caveats) relative differences in 
internal practice. But the same statistic might be 
charted overtime for each department individually 
to show how deployment has changed. Third, 
such statistics might be compared to the average 
obtained across a range of nationally- identified 
peer institutions in order to establish the position 
of each with respect to what is typical for that type 
of institution. Finally, a fixed statistic might be 
chosen as a benchmark — perhaps using an existing 
“industry standard” commonly used by an external 
agency or accrediting body. 

• technical adequacy. This criterion addresses the 
degree to which the indicator in question is reliable 
and valid as a piece of data in itself, and how robust 
it is under typical conditions of biased or missing 
data. The first of these conditions is often ad- 
vanced as the only criterion of adequacy for such 
a statistic and it is, of course, important. But 
sacrifices in data quality are sometimes justifiable 
in order to obtain a measure that is notably superior 
on other criteria. The precision of the indicator 
should be sufficient to reliably inform action, and 
real actual decisions are rarely made on the basis 
of small differences in performance. Good 
indicators should also be designed so that they can 



operate under less-than-ideal measurement 
conditions, because these will surely occur. 
Complex, large-scale surveys of former students 
and their employers, for instance, may obtain only 
marginal response rates, and the consequences of 
differing rates of response on the accuracy of the 
resulting statistics should be carefully assessed 
in advance — again, in relation to the kinds of 
decisions to be informed. 

• data availability and ease of collection. This 
final principle concerns the degree to which the 
proposed indicator is practically obtainable at a 
reasonable cost. Many otherwise promising 
indicator systems fail simply because they are too 
expensive, too complex, too time-consuming, or 
too politically costly to implement. Often, 
therefore, simple is best, even if it initially appears 
less technically attractive. Large-scale direct 
assessments of student ability, as currently 
practiced in K-12 environments for instance, 
are conceptually compelling as indicators — 
especially to constituents outside higher edu- 
cation. But designing and deploying such 
measures in postsecondary settings requires heavy 
initial resource commitments for instrument 
development and may equally require the ex- 
penditure of considerable political capital in order 
to overcome inevitable faculty resistance. Indirect 
indicators of curricular and teaching practice may 
be easier to collect and be Just as useful for guiding 
appropriate action. 

In sum, indicators work best to inform academic 
policy when a wide range of them are developed and 
when they are clearly related to particular uses and 
users. When employing indicators of any kind, 
moreover, it is important to consider the overall 
patterns that they suggest and not “zero- in” on 
small differences which may be the result of 
unimportant variations or simple chance. For 
academic administrators, systems of indicators are 
best equipped to initiate discussions about systemic 
improvement among faculty, staff and admin- 
istrators on a more concrete basis. As industry has 
shown in adopting TQM, information is most 
valuable for improving practice when it serves as a 
concrete point of departure rather than being used 
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after-the-fact to point fingers about lack of progress 
and who might be at fault. At the same time, the best 
uses of indicators for external audiences like 
policymakers or prospective students will inform 
such constituents concretely and quickly about 
issues that are important to them and about the costs, 
benefits and consequences of the particular choices 
about higher education that they will have to make. 



Good indicators can be designed to embrace both 
purposes. The best will help students, institutions, 
and policymakers at all levels recognize and take 
responsibility for their own actions and the con- 
sequences for others that these actions create. And 
as shown in many other enterprises, it is only when 
responsibility of this kind develops naturally that 
either real accountability or improvement can occur. 
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Chapter 3 



Indicators of What? 

The Domains of “Good Practice” 



T O be effective as guides for improvement, 
institution-level indicators should be or- 
ganized around a well-conceived and widely-held 
model of effective practice in undergraduate 
education. Not only does such a conceptual ground- 
ing help develop better campus understanding of the 
system’s scope and intent, but it also makes it easier 
for those collecting and using individual indicators 
to see exactly how they are intended to fit together. 
Absent an overarching model of this kind, an 
indicator system will provide little guidance for 
improvement beyond maximizing each component 
individually, however well its particular pieces are 
designed. This chapter provides one such model, 
designed to emphasize “good practice” in under- 
graduate instructional delivery. 

As attention to articulating common curricular 
outcomes increased throughout the eighties, so did 
concerns about effective instructional practices. 
Partly this was due to contemporary reforms in 
pedagogy that went beyond teaching to emphasize 
active, collaborative approaches to learning. 



Partly it was a natural consequence of wider 
assessment-for-improvement initiatives which 
quickly recognized that information about in- 
structional processes and student experiences would 
be critical in making sense of any obtained data 
about outcomes (Astin 1991). The specifics of 
instructional “good practice,” moreover, were given 
several powerful vehicles for dissemination 
throughout the eighties. A first such stimulus con- 
sisted of two well-publicized national reports — 
Involvement in Learning (NIE 1984) and Integrity 
in the College Curriculum (AAC 1985) — which 
emphasized such matters as active student learning 
experiences, frequent feedback on performance, and 
designed curricular coherence. A second influential 
document was Seven Principles for Good Practice 
in Undergraduate Education (Chickering and 
Gamson 1987), which not only succinctly codified 
many of these practices but also designed concrete 
inventories for assessing the degree to which they 
are actually found on college campuses (Gamson 
and Poulsen 1989). Growing interest in classroom 
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assessment (Angelo and Cross 1993) gave further 
impetus to these developments, as well as providing 
an additional set of “good practices” to count. 

Specific Domains of “Good Practice” 

“Good practices” within this rubric cover a range 
of instructional domains from the design and 
sequencing of curricula through the kinds of 
experiences that students actually engage in, and are 
grounded in a considerable literature on what works 
best for learning in collegiate settings assembled 
over the past three decades (Pascarellaand Terenzini 
1991). Twelve such attributes appear especially 
suitable for the development of useful indicators. 

1. High Expectations 

Research at both the K-12 and post secondary 
levels has demonstrated that students learn more 
effectively when expectations are set at high but 
attainable levels, and when expectations are clearly 
communicated from the outset. When students are 
expected to take risks and to perform at high levels, 
they make greater efforts to succeed. Absent this 
kind of encouragement, they tend to choose “safe” 
learning alternatives that allow little room for 
developing their full potentials. In contrast to more 
conventional notions of “academic rigor,” more- 
over, students should not be left on their own to 
reach set standards. Instead, both the institution and 
its faculty must actively work to help them succeed. 

From the point of view of indicators development, 
a number of specific aspects of high expectations 
appear promising. First, such expectations must be 
consistently manifested throughout the institution in 
its curriculum, environment, and teaching practices. 
In the curriculum, for instance, high expectations 
can be communicated by practices like not awarding 
degree credit for remedial or non-collegiate work, 
by clear and enforced policies on academic good 
standing, and by visible requirements that both 
syllabi and program descriptions communicate 
expected outcomes. In teaching, high expectations 
are communicated by faculty themselves in the 



form of clear statements of expected performance 
provided early in each course, by examinations and 
exercises that require students to demonstrate 
higher-order cognitive skills rather than simple 
factual recall and by active efforts to help students 
attain established goals. In the wider academic 
environment, high expectations are communicated 
by campus activities that demonstrate and reward 
academic excellence, by policies that encourage 
students to enroll in advanced courses outside their 
major fields, and by policies that encourage students 
to fmish the courses in which they initially enroll and 
to persist in general. 

2. Coherence in Learning 

Students also succeed best in developing valued 
higher-order skills when such skills are reinforced 
throughout their educational programs. This means, 
at minimum, that students’ learning experiences 
should consist of more than merely a required 
number of courses or credit hours. Instead, the 
curriculum should be structured in a manner that 
sequences individual courses to reinforce common 
outcomes and that consciously directs instruction 
toward collective ends. 

More specific aspects of coherence suggest some 
particular directions for developing useful indi- 
cators. In curricular design, coherence is manifested 
in required course structures and sequences, 
particularly in general education. To achieve 
coherence, courses should be intentionally inter- 
related in coverage and expectations, with key 
learning objectives built in throughout. In teaching 
practice, moreover, faculty should encourage 
students to actively make connections among 
courses, disciplines, and fields of knowledge — 
either by systematically using material from other 
courses and settings as examples or by encouraging 
students to create projects that relate two or more 
courses. Finally, the campus environment itself 
should encourage coherence by providing multiple 
opportunities for students to make connections 
among major academic themes — for example, 
through lecture series or cultural events. 
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3. Synthesizing Experiences 

Students also learn best when they are required to 
integrate knowledge and skills. Typically where 
this occurs in baccalaureate curricula, it involves a 
senior seminar, project, external examiner, or thesis 
that requires a synthesis of knowledge and a demon- 
stration of independent inquiry or application — a 
curricular feature that generally occurs (if at all) at 
the end of a major program. But such experiences 
are appropriate at multiple points in a student’s 
career, and should not be confined to either 
upper-division or baccalaureate programs. 

Following this logic, specific indicators might 
fiiiitfiilly be based on a number of features. In 
curriculum design, the existence of “capstone” or 
integrative experiences as a requirement for grad- 
uation often suggests institutional commitment to 
this principle. In teaching practice, moreover, the 
principle is reflected in the degree to which faculty 
visibly require students to demonstrate the ability to 
connect key concepts and relate appropriate skills. 
Finally, in the wider institutional environment, 
students should actively seek such opportunities for 
synthesis in projects that they undertake on their 
own. 

4. Integrating Education and Experience 

Classroom experiences are both augmented and 
reinforced by multiple opportunities to apply what 
is learned. In professional curricula, opportunities to 
actively apply skills in this manner are often the 
norm, and include formal practice, internships, or 
cooperative education arrangements. But such 
opportunities are generally lacking for more 
traditional disciplines or for undergraduate edu- 
cation as a whole. 

Particularly suitable for guiding indicators 
development here are curricular designs that require 
all students to complete an internship or similar 
experience as an integral part of their programs. 
And rather than being entirely independent 
activities, these experiences should be directly and 
formally related to classroom work. Where appro- 



priate, curricula consistent with this principle also 
allow degree credit to be awarded for planned, 
education-related work experience. In teaching 
practice, faculty should refer consistently to 
work-related or practice-oriented applications of 
their subjects. Often, the best examples of this occur 
when faculty “model” their own work in the 
classroom. Work-related practice can fiirther be 
reinforced by rethinking policies regarding student 
assistance — emphasizing, for example, the role of 
college work-study as an alternative to more con- 
ventional forms of financial aid. Finally, the 
institutional environment itself should provide 
multiple opportunities for students to engage in 
education-related work, even when there is no 
programmatic requirement that they do so. 

5. Active Learning 

At all levels, students learn best when they have 
multiple opportunities to actively deploy their skills. 
Rather than relying exclusively on instructional 
settings that emphasize passive listening in 
introductory classes, this principle implies frequent 
discussion of presented class material, considerable 
written work, and the application of learned material 
to new settings or contexts. At the same time student 
assessments, rather than being based entirely on 
information recall, should require active demon- 
stration of synthesis and application. 

These attributes suggest a number of possibilities 
for developing indicators. In curricular design, they 
imply that a significant number of classes should be 
small or be structured in a manner that allows active 
interchange to occur among students and faculty. In 
the classroom, faculty should use pedagogical 
techniques that require students to become actively 
engaged, such as structured discussion, frequent 
writing, independent projects or other applications- 
related exercises and information-gathering tasks. 
Finally, the institutional environment should pro- 
vide multiple opportunities for informal inter- 
change to occur among students in discussing class 
material, as well as between students and faculty 
members. 
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6. Ongoing Practice of Learned Skills 

A common research finding in both K-12 and 
postsecondary study is that unpracticed skills 
quickly atrophy. This is especially the case with 
core skills like computation and writing which, if 
not reinforced, will inevitably deteriorate. Good 
practice consistent with this principle requires 
the provision of multiple opportunities for students 
to exercise such valued higher-order skills as 
communication (written and oral), critical thinking 
and problem-solving, or basic quantitative tech- 
niques. It also requires that students demonstrate 
such skills at appropriate levels as a condition of 
graduation. 

Indicators consistent with these principles center 
particularly on attributes like the following. First, 
curricular designs should reflect a need to practice 
and reinforce identified core skills throughout the 
curriculum, instead of the development of particular 
skills being made the sole responsibility of 
specific departments or courses (e.g., assuming that 
an English Composition course, or the English 
Department in general, is entirely to blame if 
students graduate without valued writing skills). 
Furthermore, curricular requirements should be 
designed so that students cannot avoid classes that 
enhance or develop these skills. Second, faculty 
must require a significant amount of such practice in 
classroom assignments — especially in writing and 
quantitative reasoning. Indeed, the notion of 
practice of this kind should be internalized as a 
primary element for certifying mastery. Finally, the 
institutional environment in general should foster 
student exercise of basic academic skills through 
clubs and associations, discussion and study groups, 
and similar activities. 

1 . Assessment and Prompt Feedback 

Frequent feedback to students on their own 
performance is also a major contributor to learning. 
Typically in college classrooms, students receive 
little formal feedback on their individual work until 
well into an academic term. Consistency with this 
principle demands that institutions pay far greater 
attention to providing students with information 



about their own performance — both within formal 
coursework and through advisement and other 
experiences that give students an opportunity to 
assess more broadly what they have learned. At the 
same time, early classroom assessment allows 
faculty to determine the many levels of student 
ability and variety of backgrounds that may be 
present in a given classroom, and can suggest some 
strategies for dealing with this diversity. 

An especially wide range of specific mani- 
festations of this “good practice” is suitable for the 
development of indicators. In curriculum and 
academic policy, consistency with this principle 
demands early assessment of students’ abilities on 
entry to college and an active effort to remediate 
identified deficiencies. In course and curricular 
designs, students should be required to demonstrate 
their skills in short exercises throughout the term, 
rather than only on a mid-term or final examination. 
In teaching practice, consistency with this principle 
demands quick “turnaround” of assignments, and 
assessments of student performance that identify 
individual weaknesses and how these might be im- 
proved. At the same time, through such techniques 
as classroom research, faculty should actively 
monitor how the class as a whole is progressing and 
what needs greater emphasis, and make modifi- 
cations accordingly. Finally, in the wider environ- 
ment, students should be encouraged to take classes 
that will help them to strengthen weak areas, not 
just reinforce areas in which they are already 
strong. Similarly, the institutional environment 
should encourage student self-assessment at every 
level — allowing students multiple out-of-class 
opportunities to test their skills through peer 
assessment (e.g., through the use of study groups), 
or by providing facilities like self-paced learning 
laboratories or study centers. 

8. Collaborative Learning 

Students learn better when they are engaged in a 
team effort rather than being entirely on their own. 
Working with others increases involvement and 
provides multiple opportunities for feedback. At 
the same time, it reinforces communication and 
problem-solving skills. At least as important in 
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fostering skills of working together is modeling how 
the world outside the classroom works — a world 
increasingly based on teams and work groups that 
students will soon face. Research also suggests that 
collaboration is a useful model for faculty/student 
interaction. Rather than being solely a judge of 
student performance, the best teachers act as 
“coaches,” working with students as joint par- 
ticipants in achieving a common goal. 

Useful indicators can be developed for many 
specific aspects of this domain. In the realm of cur- 
riculum, courses and programs should be designed 
to incorporate group projects accomplished both 
within and outside of class, and classes themselves 
should be structured to promote such interaction. At 
the same time, instructors should use pedagogical 
techniques that require students to interact with one 
another, often in the context of group research 
projects or problem-solving exercises. Finally, the 
campus environment should be organized to 
foster and sustain informal collaborative activity 
in academic areas such as study groups and 
academically-related clubs and organizations. 

9. Significant Time on Task 

Not surprisingly, research also confirms that 
greater amounts of time invested in learning yield 
greater payoffs in terms of what and how much is 
learned. Consequently, how an institution defines 
its expectations of the ways students and instructors 
should use their time can powerfully influence the 
quality of learning that occurs. At the same time, 
visibly emphasizing time on task helps students 
learn how to plan more effectively, to manage their 
time, and to focus their energy. 

Time on task is especially well-suited to the 
development of useful indicators. In the curriculum, 
consistency with this principle suggests course 
designs that allow adequate time for practicing skills 
(for example, laboratories and workshops) in 
addition to the time set aside for more formal 
instruction. It also suggests more directive 
approaches shaping students’ use of out-of-class 
time to achieve educational objectives. Similarly, it 
suggests that faculty should themselves devote 
significant time to the task of undergraduate in- 



struction including (in addition to classroom 
instruction and preparation) substantial office hours, 
out-of-class contact with students, and scholarship 
related to teaching in their disciplines. Finally, the 
campus environment should be deliberately struc- 
tured to enhance greater time on task by encouraging 
students to spend out-of-class time on academic 
rather than social and other pursuits. Study spaces 
should be available and well-used in the library, 
residence halls and other public areas, and there 
should be mechanisms available such as living/ 
learning communities to facilitate group study and 
other collaborative efforts. 

10. Respect for Diverse Talents and 
Ways of Knowing 

Students come to college with vastly different 
backgrounds, levels of preparation, and kinds of 
previous experience. And regardless of background, 
different students may learn effectively in quite 
different ways. Good practice demands carefully 
designing curricula and instructional practices to 
meet this expected variety. Diversity, moreover, 
can itself be of instructional value. Not only should 
individual ways of knowing be respected and 
students be allowed to capitalize on their individual 
strengths, but diversity should also be harnessed for 
the insights it can provide on the subject matter being 
taught. Instructional approaches that actively tap 
prior student and faculty experiences, and that 
highlight the differences in these experiences, can be 
particularly effective. 

Many specific aspects of this domain suited for 
indicators development are available. In the 
curriculum, consistency with this principle means 
that students should be able to create programs that 
play to their strengths, especially in lower-division 
courses. One implication may be that the student’s 
first year not be devoted solely to conceptual 
introductory coursework, but contain also the kinds 
of “hands-on” courses typically taught later in a 
curricular sequence. At the same time, faculty 
should focus on teaching methods that are receptive 
to a full range of student learning styles and 
operations — for instance, emphasizing practical 



21 



applications in conceptual courses and emphasizing 
connections with theory in applied courses. Finally, 
the campus as a whole should actively support 
diversity in both its programs and its behavior. 
Students of diverse backgrounds ought to interact 
actively and informal mechanisms should be 
available for students to learn from one another. 

11. Frequent Faculty-Student Contact 

Considerable research suggests that frequency of 
out-of-class academic contact between faculty and 
students is strongly associated with both program 
completion and effective learning. Knowing a 
few faculty members well enhances students’ in- 
tellectual commitment and encourages them to think 
about their own values and future plans. At the same 
time, through such contact students are able to see 
faculty less as experts than as role models for 
ongoing learning. 

Developing indicators of frequency of faculty- 
student contact appears promising in a number of 
areas. In the curriculum, it can be enhanced by 
designs that emphasize occasions for contact as well 
as simply time on task — for example, scheduling 
three one-hour class meetings in preference to one 
three-hour session. In parallel, classes should be 
small enough or be explicitly structured to promote 
informal interactions between instructor and 
student. Faculty, in turn, ought to have oppor- 
tunities to interact more frequently with students in 
such out-of-class settings as learning communities 
or recreational activities. Finally, the campus 
environment should be structured so that informal 
interaction is actively encouraged throughout the 
campus in the design and location of faculty offices 
and through the provision of informal meeting and 
study spaces in residence halls and other public 
areas. 

12. Emphasis on the Early Years of 
Study 

Consensus is emerging that the first years of 
undergraduate study — particularly the freshman 



year — are critical to student success. Partly this is 
because the transition from high school to post- 
secondary study represents for most students a 
major discontinuity in both expectations and 
behavior. Not only are standards typically higher, 
but students also are expected to work independently 
and to make significant independent choices about 
their individual courses of study. For adult students 
returning to the unfamiliar world of postsecondary 
study after many years, the shock of transition can 
be particularly abrupt. Yet the pattern of resource 
investment at most colleges and universities 
strongly favors upper-division work. At the same 
time, comprehensive efforts to integrate first-year 
students into the mainstream of the collegiate 
experience are often treated as auxiliary enterprises, 
unconnected to the faculty and to core academic 
experiences. 

Emphasis on the early years can be manifested 
specifically in a number of ways. In the design of 
the curriculum, such emphasis implies re-examining 
introductory courses to make them more than simply 
first courses in an anticipated disciplinary sequence 
in the major. Similarly, typical curricular designs 
that emphasize rigid progression from general 
education in the first two years to in-depth study 
should be critically examined to ensure that students 
are not being lost because they are forced to take 
courses that they see as irrelevant to their goals. 
Faculty, in turn, should be invested more heavily in 
the first years of study. This means not only staffing 
introductory courses with regular (and preferably 
senior) faculty, but also more fully involving 
departmental faculty in freshman-year advising and 
experience programs. Finally, ensuring the success 
of first-year students should be visibly stressed as a 
campus-wide priority. Targeted activities such as 
Freshman Year Experience Programs should be 
consistently reinforced by a campus environment 
in which all personnel — academic and non- 
academic — are demonstrably committed to the 
common goal of improving first-year student 
retention and performance. 
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Some Cross-Cutting 

Indicator Domains 

“Good practices’’ of these kinds can be manifested 
in many different ways in a given institution or 
program. As Stark and others (Stark and Lowther 
1986) have pointed out, institutions may possess 
multiple “curricula” operating in parallel that may 
have little to do with one another in content, 
coverage, or effectiveness. For purposes of indi- 
cators development, it is often useful to conceive 
of these distinct “curricula” systematically to 
determine specifically whether or not particular 
features are present. Doing so results in some 
additional domains for guiding indicators devel- 
opment, which cut across the kinds of “good 
practices” described in the previous section. 

* institutional and program requirements. In- 
dicators in this class address the degree to which 
curricular and other educational requirements 
contain particular structural features. Con- 
sequently, they address what can be called the 
“designed curriculum,” consisting primarily of 
catalogs and syllabus descriptions. This par- 
ticular “curriculum,” of course, tends to dominate 
faculty attention, regardless or whether it has 
reality in practice. While increasingly recognizing 
explicit learning outcomes as part of the design, 
the bulk of attention remains focused here on 
course content and sequence as defined by 
institutional or departmental policy. 

A deeper and less commonly-undertaken look at 
curricular design, moreover, can be labelled the 
“expectational curriculum.” This consists of the 
specific assignments and levels of performance 
expected of students and the manner in which 
student performance is assessed. At the same 
time, it reflects what is typically the student’s view 
of a given curricular design — based not on course 
content but rather on the specific requirements 
that must be met and the levels of performance 
needed to meet them. Examining expectations in 
this manner can be highly revealing; a stated 
objective of the general education curriculum, for 



instance, may be to develop students’ oral 
communications proficiency, while an exam- 
ination of actual required performance across 
general education courses reveals that students are 
rarely asked to demonstrate this proficiency and 
are never evaluated on it. 

• instructional delivery. Indicators in this class are 
designed to address the degree to which particular 
instructional behaviors and settings actually occur, 
regardless of the original design. Consequently, 
this domain addresses what can be called the 
“delivered curriculum,” consisting of what faculty 
actually teach and the consistency with which they 
do so. In contrast to design, this “curriculum” is 
behavioral, and may vary significantly across 
classrooms and from original design speci- 
fications. In addition, for any established 
curriculum, the phenomenon of “course drift” 
occurs. As new faculty teaching assignments to a 
given course are made, both content and teaching 
methods may vary increasingly from the vision 
embodied in the original design. Similarly, 
significant unintended variation may occur from 
section to section in a multi-section course. 
Typically, institutions know very little about the 
extent and impact of such variations beyond 
simply acknowledging that they occur. 

• student experiences. Indicators in this class 
address the degree to which students actually 
engage in particular learning behaviors — both in 
and out of class — ^that are associated with par- 
ticular outcomes. As such, it is centered in what 
can be termed the “experienced curriculum,” 
consisting of what students really do. This 
curriculum also is behavioral and can usefully 
be divided into two quite different components. 
One concerns student coursetaking patterns — 
especially when these occur within a given dis- 
tributional or elective design. Formal structure 
may prescribe particular arrays of courses that 
meet specific requirements, but how do different 
kinds of students in fact act out these requirements 
in the actual course choices that they make? A 
second aspect of the experienced curriculum is 
student learning behaviors. How much writing do 
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students typically complete, for instance, in order 
to meet class assignments, regardless of what a 
syllabus analysis of requirements might reveal? 
Or, consistent with the example above, how much 
actual exercise of oral communications skills do 
they report as having occurred in the classes that 
they take? Again, institutions typically know very 
little about such matters though they often de- 
vote meticulous attention to up-front issues of 
curricular design. 

All three aspects of curriculum are important to 
assess, and appropriate indicator systems can be 
designed to reflect them all. Indeed, the best 
such systems are especially configured to detect 
discontinuities among them — for example, ex- 
amining the ways in which particular elements of 
intended design are not being effectively translated 
into teaching practice and student experience. 
Detecting discontinuities of this kind, moreover, 
may be especially important when the elements of 
design in question correspond to the kinds of “good 
practices” described in the previous section. Many 
sound curricular approaches intended to reflect such 
practices were deployed in colleges and universities 
throughout the eighties. But fi*equently they were 
not well implemented, and both faculty and students 
continued behaving much as before. This suggests 
that indicators be consciously designed to in- 
corporate both the presence and character of 
intended “good practices” as well as the degree to 
which they are “acted out” in the form of appro- 
priate policies and behaviors. 

Applying this approach yields the taxonomy 
shown in Figure 2, which can be used as a rough 
guide for both developing individual indicators and 
for assessing their collective coverage. Cells within 
this matrix consist of particular intersections 
between particular “good practices” on the one hand, 
and how they specifically are present in institutional 
expectations, instructional delivery, and student 
experiences on the other. An additional column in 
the matrix addresses the degree to which resources 
(fiscal, human and physical) are allocated in support 
of known “good practices,” and constitutes a 
distinctive domain for developing indicators. The 
catalog of sample good practice indicators contained 



in this Handbook is based on this taxonomy, and 
each entry in the catalog references a particular set 
of cells within the matrix. 

Indicators and Outcomes 

When developing and using indicators based 
largely on “good practices,” the question of the 
relationship of such practices to actual learning 
outcomes frequently arises. Many maintain that 
directly measuring educational outcomes provides 
much more valid and credible evidence of in- 
structional quality than examining educational 
processes, and that assessments of this kind should 
be preferred wherever possible. Up to a point, 
this argument is reasonable: indicators of “good 
practice,” no matter how well developed, cannot 
substitute for outcomes measures. The principal 
premise of this Handbook is that such indicators are 
useful in their own right and can be valuable in both 
guiding practice and improving accountability. 
Developing “good practice” indicators that are 
substantively related to outcomes is quite feasible, 
and what research tells us about the empirical 
strength of these relationships should be carefully 
considered. 

A first point to consider is that all indicators of 
educational attainment are in some sense “indirect.” 
Cognitive measures of student attainment like stand- 
ardized tests and performance-based assessments 
are no exceptions, and should not be confused with 
the actual entities that they purport to represent. But 
less direct representations of these abilities for in- 
dicative purposes may be of two very different kinds. 
On the one hand, useful and reliable statistics for 
tracking institutional progress in improving instruc- 
tion might be developed that are highly correlated 
with desired outcomes without being causally re- 
lated to these abilities. Common examples here in- 
clude the results of other, more easily administered, 
examinations known to be related to the traits in 
question or student self-reports of gains in skills or 
knowledge. As noted in Chapter 2, proxy indicators 
such as these are commonly used in other fields to 
document trends, but their major drawback is that 
they cannot generally be used to inform policy or to 
direct action. On the other hand, indicators can be 
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Figure 2 



A Taxonomy for Developing “Good Practice” Indicators 

A BCD 

Resource Institutional Instructional Student 

Investments Requirements Delivery Experiences 

1 High Expectations 

2 Coherence in Learning 

3 Synthesizing Experiences 

4 Integrating Education 
and Experience 

5 Active Learning 

6 Ongoing Practice 

7 Assessment and 
Feedback 

8 Collaborative Learning 

9 Time on Task 

10 Respect for Diversity 

1 1 Student/Faculty Contact 

12 Early Years of Study 
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developed that reflect an underlying philosophy that 
knowledge of educational inputs and processes is 
worth obtaining in its own right, so long as evidence 
exists that these are broadly related to cognitive 
attainment. Here the payoff for academic manage- 
ment is not only the ability to monitor progress but 
to inform intervention as well. 

The latter philosophy, of course, is the one that 
grounds this Handbook. Evidence of its appro- 
priateness is based on a focused review of 
the empirical research literature on general 
collegiate skills attainment previously conducted by 
NCHEMS ( 1 994) under contract to the U. S. Depart- 
ment of Education, intended to assess the feasibility 
and utility of developing a range of “good practice” 
indicators that might be suitable for collection on a 
national basis. In this review, three specific 
informational domains were examined for their 
linkages with outcomes. A first review of in- 
structional requirements covered the relationship 
between outcomes and specific curricular re- 
quirements or coursetaking patterns, particular 
instructional designs, and expected levels of student 
performance. A second review of instructional 
practices covered similar associations with class size 
and structure, specific classroom activities and 
behaviors, and influences attributable to the wider 
institutional environment. A third review of student 
experiences and behaviors covered relationships 
between outcomes and student time on task, quality 
of effort, and overall involvement. 

Results of these reviews are summarized in Figure 
3. As shown, the strongest empirical relationships 
were noted for “active learning” classroom prac- 



tices, for broad levels of student involvement in the 
institutional environment, and for high student time 
on task and quality of effort. More specifically, the 
overall results of this exercise are sufficiently 
strongly-patterned to suggest far stronger linkages 
between desired outcomes and what happens in class 
and what students do, than it suggests the importance 
of particular institutional investments or curricular 
structures. Where the latter are important, more- 
over, it appears to be largely because of the 
opportunities for actions and behaviors that they 
enable, and not so much because these features are 
important in themselves. A brief additional review 
of evidence for the validity of self-reported cog- 
nitive development through student questionnaires 
was also undertaken as part of this exercise. As also 
noted in Figure 3, results suggested moderate but 
reliable associations between self-reports and di- 
rectly measured levels of cognitive attainment. 

In assessing the utility of sample indicators 
contained in this Handbook's catalog — or in de- 
veloping additional “good practice” measures of 
their own — institutions should carefully consider 
the strength of these associations. Where the 
existence of a strong relationship between cognitive 
attainment and the indicator being considered is 
important, such evidence should be prominently 
considered in making a choice. In many cases, 
however, the existence of an empirical relationship 
of this kind may be less important. Levels of student 
satisfaction or student/faculty ratios, for example, 
may be worth monitoring because they indicate 
phenomena of high interest to external con- 
stituencies regardless of any direct relationship with 
student outcomes that may be present. 
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Figure 3 
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Chapter 4 



Data Sources for ‘‘Good Practice” Indicators 



I ndicators of instructional practice can be derived 
from many sources and the best such systems rest 
on a range of data-collection methods. As might be 
expected, some data sources are better suited than 
others for providing data on particular domains. In 
conjunction with the catalog of “good practice” 
indicators included in this Handbook^ this chapter 
briefly describes some of the data sources most 
commonly used. At the same time, it suggests some 
ways of presenting such information to various 
constituencies, and of organizing and archiving such 
data designed to maximize their utility for academic 
planning and decisionmaking. 

Types of Data Sources 

The best indicator systems employ data sources 
that deliberately reinforce one another. “Hard” data 
on student coursetaking drawn from actual transcript 
records, for instance, can be used to cross-check 
student self-reports on the range of courses that they 
have taken or the kinds of experiences that they 
have engaged in. Indeed, triangulation of this kind 
remains one of the best safeguards against biased or 



misleading conclusions drawn from evidence which 
is by nature indirect. 

Among the most prominent sources of data 
generally used to construct indicators of “good 
practice” are the following: 

• catalog or syllabus review. Probably the most 
straightforward method for developing useful 
indicators of intended content and coverage uses 
published descriptions of courses. Applying this 
approach at the broadest level, indicators can be 
constructed about the overall structure and 
coverage of instruction for the institution as a 
whole and across its programs. Catalogs and other 
documents that include descriptions of the specific 
courses needed to complete requirements and the 
sequence in which these must be taken, for 
instance, can yield useful measures of curricular 
coverage and coherence, of disciplinary concen- 
tration, or of the relative amount of choice among 
courses allowed to students in meeting program 
requirements. At a somewhat deeper level, 
individually-prepared faculty syllabi can provide 
additional data about the ways in which instruction 
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is actually being delivered. Tapping the rich array 
of individual syllabus material typically available 
on any campus, for instance, allows detailed 
examination of the “expectational” curriculum: 
how much writing do faculty actually expect 
students to accomplish in their courses, for 
example, regardless of published goals or ad- 
vertised program requirements? Syllabus reviews 
also allow measures of conformity to objectives 
to be constructed: for each outcome or goal of 
general education, for instance, how many courses 
available for general education credit actually 
address this intended outcome explicitly as a 
course goal? At the highest level of detail it is 
possible to match designed course material to 
explicitly-identified levels of competency on in- 
tended core outcomes in order to physically map 
the specific places within a particular curricular 
sequence at which these levels are developed, 
required, and actually assessed (Ewell 1996). 

• transcripts or student records. Transcript rec- 
ords provide the most comprehensive basis on 
which to develop measures of the behavioral 
curriculum. Unfortunately, they are also one of 
the most intractable sources of information 
available, because student records systems at most 
institutions are not usually designed to analyze 
patterns of student coursetaking. Fortunately, 
some progress in tapping this rich source of 
information has occurred. First, useful national 
standards for coding samples of transcripts that 
allow meaningful comparisons of coursetaking 
patterns across quite different types of programs 
and settings have been developed (Adelman 1990, 
1995). Using these standard coding categories, 
measures of such phenomena as student exposure 
to particular fields of study can be developed — 
which often differ substantially from formal 
program structures. Second, some powerful 
summary measures of overall coursetaking 
behaviors have been constructed that reduce 
extremely complex transcript detail into broad 
behavioral patterns. These include, for example, 
overall depth of disciplinary exposure or the 
degree of course choice typically exercised by 
students drawn from different demographic 



groups or enrolled in different programs of study 
(e.g., Zemsky 1989). Such measures are useful 
not only as indicators in themselves, but also as 
experiences to be associated with student 
performance. Finally, transcript files allow some 
measurement of performance in the form of grades 
earned or subsequent courses passed. If indivi- 
dual assessment results are also included in an in- 
stitution’s record system (or can be matched to 
individual student records via an identification 
number), powerful indicators can be constructed 
about the adequacy of prior curricular preparation 
or about the contribution of particular courses to 
specific outcomes. Detailed methods for con- 
ducting analyses combining student transcript 
records with assessment data are available from 
several sources. One example is the “differential 
coursework methodology” developed by the 
National Center on Postsecondary Teaching, 
Learning and Assessment at Penn State (Ratcliff 
and Associates 1988). Another is the approach 
used by Astin (1992) to identify specific curricular 
experiences to be linked to student gains in 
achievement. In both these cases, a standard 
coding scheme is used to classify the individual 
courses contained in a given student’s transcript 
record, the results of which are then linked to 
additional data on experiences or outcomes. 

• self-reports from faculty and students. What 
faculty say they do and value and what students 
say is delivered or experienced are eimong the most 
powerful ways to get at the behavioral aspects of 
instruction. Indicators of curricular quality based 
on student self-reports, for instance, are ideally 
suited to assess “good practice.” While syllabi and 
transcript records can record levels of expectations 
and patterns of attendance far more accurately 
and completely than can student recollections, 
students themselves are probably in the best 
position to report on such matters as their own 
levels of effort, whether or not they spoke in class 
and how frequently, or the degree of group-work 
that they might have engaged in. 

Most institutions already use survey self- 
reports to evaluate individual classes through the 
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familiar mechanism of student ratings of 
instruction. But such instruments are generally 
aimed at a single purpose: evaluating the beha- 
vior of individual instructors for purposes of 
promotion and tenure. Increasingly, however, 
such questionnaires are being reconfigured to 
additionally capture students’ own behaviors or 
perceived learning (Ewell 1991). This trend has 
been given a powerful impetus by the classroom 
assessment movement (Angelo and Cross 1993), 
but remains distinct in purpose because the results 
of classroom research are not generally intended 
to be codified or widely shared across the insti- 
tution. An important caution involved in using 
information drawn from such sources, therefore, 
is to ensure a proper level of aggregation in 
reporting the resulting indicators. It is generally 
inappropriate, for example, to report student 
testimony on such matters on a class-by-class or 
an instructor-by-instructor basis. 

While items designed to assess time on task, 
amount of writing assigned and accomplished, or 
the incidence of active learning experiences are 
well-suited for inclusion in end-of-course ques- 
tionnaires, they are equally amenable to broader 
questionnaires administered periodically to 
samples of students. Most institutions make 
occasional use of student surveys of this kind, but 
most concentrate on overall student satisfaction 
and student use and reactions to specific services. 
Few such questionnaires include items especially 
tailored to tap learning experiences and behaviors 
known to be related to achievement. The 
indicators catalog included with this Handbook 
contains a wide variety of questionnaire items of 
this kind, which institutions can use in con- 
structing their own instruments. The most 
prominent national examples of instruments 
that embody such items are the Cooperative Insti- 
tutional Research Program (CIRP) Follow-Up 
Survey (Astin and Associates 1992) administered 
to nationally-normed samples of college students, 
and the College Student Experiences Ques- 
tionnaire (Pace 1984) now available through the 
University of Indiana. The latter is especially 
suited to tapping many of the dimensions of good 
practice described in this Handbook, and new 



scales have been developed to explicitly reflect 
three of them — student/faculty contact, active 
learning, and cooperation among students. 

Reports from faculty constitute a similar, but 
less well-tapped source of information about 
instructional behaviors. Some surveys of this 
kind have been designed to document typically- 
used instructional practices — for instance, the 
model instruments designed to assess faculty 
application of the “Seven Principles of Good 
Practice in Undergraduate Education” (Gamson 
and Poulsen 1 989) or the surveys used by Angelo 
and Cross (1993) to investigate the perceived 
importance of particular goals for instruction in 
faculty teaching behavior. Others are more 
oriented toward determining broader underlying 
instructional values (for example, Astin 1992). 
Probably the most common indicators collected in 
this way are amounts of time actually spent on 
instruction and reported use by faculty of 
particular instructional techniques. 

Whether self-reports are obtained from faculty 
or students, questions of validity often arise. These 
are generally of two kinds. One is the overall 
credibility of self-reported behaviors in particular 
performance situations. Faculty, for instance, are 
more likely to provide accurate testimony about 
the general value placed on teaching in their 
departments than they are to reliably report their 
own behaviors. Similarly, students are usually 
better judges of what they have done (e.g., how 
they spent their time) than they are of how much 
they have learned. A second difficulty of using 
survey data to develop indicators is the fact that 
data drawn from questionnaires rarely yield 
usable “point estimates” of performance — 
especially on items related to attitudes or 
satisfaction. A finding that 58% of students 
majoring in biology are “extremely satisfied” with 
instruction in their major, for instance, is far less 
useful in itself than knowing that this value was 
76% for last year’s graduating class, or that a 
difference of 15% in satisfaction levels separates 
the men from the women. As a result, the best uses 
of indicators drawn from survey sources rely 
strongly on analyses of trends over time, or 
of the differing experiences of particular sub- 
populations of students and faculty. 
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Self-reports about specific outcomes and ex- 
periences, on the other hand, can be unusually 
credible when used to provide “consumer 
information” to outside audiences. Policymakers 
and potential students, for instance, will likely 
consider the testimony of employers and former 
students to be considerably more valid and 
relevant in informing the specific choices that they 
must make than far “harder” data supplied by 
institutions that lack the perspective of direct 
experience. 

* assessment and student work samples. Direct 
assessment of student achievement often provides 
the most convincing evidence of instructional 
effectiveness. As a result, most institutions are 
currently engaged in some kind of outcomes 
assessment effort. Increasingly, however, the 
assessments engaged in are “authentic” — relying 
less on purpose-built measures such as standar- 
dized tests and comprehensive examinations and 
more on embedded assessment devices that allow 
broad information about student development to 
be inferred from re-examining actual classroom 
work. Among the most popular approaches are 
student portfolios and work samples. Here 
institutions assemble typical examples of student 
work produced as part of the instructional process, 
then score these naturally-occurring products 
according to pre-designed criteria to derive a 
measure of such abilities as writing, critical 
thinking, or awareness of cultural diversity. 

Quite properly, these techniques have up to now 
focused principally on the two extremes of the 
teaching-learning continuum — determining 
overall levels of institutional or program 
attainment in relation to desired outcomes, and 
guiding the process of instruction within indi- 
vidual classrooms. But outcomes assessments 
such as these can play two additional roles. One 
is the often considerable amount of information 



about real curricular content that can be obtained 
through the use of such assessments. Long-time 
observers of assessment in action have parti- 
cularly emphasized the indirect benefits of the 
process in clarifying goals (Banta and Schneider 
1988, Banta 1993). Others have noted how 
assessment evidence of this kind often has as 
much (or more) to say about curricular structures 
as it does about actual outcomes (Ewell 1991). 
For example, institutions using portfolios have 
occasionally found this exercise more valuable in 
determining what faculty are really assigning, and 
the alignment of these assignments with curricular 
goals, than in directly documenting student 
attainment (Ewell 1996). 

A second underattended application of assess- 
ment techniques is to determine more precisely 
whether intended connections among courses are 
in fact occurring. Considerable evidence on this 
topic can be compiled by examining transcripts 
and other student enrollment records. But more 
direct methods are also available for determining 
the degree to which previously-learned content 
and skills are effectively deployed in new con- 
texts. One such approach is to embed pre-tests and 
post-tests directly into specific course sequences, 
designed especially to determine if prerequisite 
skills present at the end of one course are in fact 
present in the next (Farmer 1988). A variant on 
this approach is to systematically code the kinds 
of errors students make in a subsequent course 
when applying previously-covered material 
(Harris and Baggett 1992). 

As these examples suggest, the most powerful 
approaches to assessing good practice will employ 
both outcomes and process measures. In developing 
appropriate indicator systems for purposes of either 
external accountability of internal curricular 
improvement, institutions are well advised to pay 
equal attention to both. 
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Developing a “Quality 
Management Database” 

Much of the data required to construct appropriate 
indicators of instructional “good practice” is already 
collected by institutions. But the data required are 
not usually maintained or configured in a manner 
that allows their easy retrieval and manipulation. 
Needed data elements are often located in different 
computer files or records systems (if, indeed, 
electronic records exist at all). To create usable 
indicators, they must often be extracted laboriously 
from these sources and reassembled in a more 
suitable medium — generally a spreadsheet or a 
microcomputer database. This is usually a difficult 
and time-consuming task, and one that must be 
repeated each time such a statistic is wanted. In 
particular, modifying current transaction-oriented 
computerized record systems such as registration or 
fmancial systems — ^which are designed for a quite 
different purpose — ^to construct appropriate indi- 
cators, may require extensive re-design and re- 
programming. 

Similarly, the results of “special-purpose” data- 
collection efforts like student surveys and outcomes 
assessments are typically maintained in separate 
data files by the offices that collect them. While 
such data are usually amenable to retrieval and 
further manipulation, they may be stored in quite 
different formats and they are rarely documented in 
a manner that allows access beyond those who 
originally collected and analyzed the data. More 
importantly, most such data-collection efforts 
contain far more data than may be relevant in 
producing indicators for general institutional 
purposes. A questionnaire administered annually to 
all enrolled students by the Office of Student Life, 
for instance, may contain only two or three items of 
interest — for example, student perceptions of the 
supportiveness of the campus environment to 
working in teams — ^while the vast majority address 
topics that are only relevant to the office ad- 
ministering the survey. General-purpose data items 
such as these may be contained in dozens of 
unrelated surveys and assessments, but unless they 
are centrally assembled and adequately documented 



it may be simply too much trouble to access them 
individually when developing indicators. 

Given these conditions, many institutions have 
found it beneficial to develop a set of specially- 
designed analytical databases configured specifi- 
cally to generate regularly-used performance 
statistics like the indicators described in this 
Handbook. Data elements contained within this 
database are periodically extracted from operational 
records systems and from any regularly-occurring 
surveys and assessments at defined census intervals. 
And the file structures of the database itself are 
configured to allow easy calculation of statistics and 
performance ratios drawn from quite different data 
sources. 

Because such files are configured as integrated 
relational databases, manipulating the information 
to create needed performance measures and to 
conduct typical ad hoc queries is straightforward. 
But it is critical to design the management database 
in this fashion from the outset, rather than 
developing calculational routines for individual 
indicators in isolation. Even if the fiill database 
design is never fully realized, the systematic 
thinking about information that it provokes can 
provide a major gain for the institution in linking and 
interpreting the data that are available. Indeed, 
many institutions have found that one of the major 
benefits involved in putting such a design together 
is that it forces institutions to inventory (and perhaps 
re-structure) the often considerable array of data on 
instructional delivery that they already have (Ewell 
and Lisensky 1988). 

Figure 4 outlines the basic features of an academic 
“quality management” database of this kind by 
contrasting it with “transaction-oriented” records 
systems on the one hand, and top-level “decision- 
support” systems on the other. As noted in the 
figure, most colleges and universities already have 
many “operational data systems” designed primarily 
for record-keeping purposes, that reside on admin- 
istrative computers or in individual unit-level 
microcomputers. These are designed primarily to 
provide an automated capability for managing 
individual transactions (such as cutting a check 
or generating a transcript) or for retrieving and 
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Figure 4 
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inspecting individual records (such as supporting an 
advising session or making an individual admissions 
decision). Requirements for decision support, at the 
other end of the information spectrum, are very 
different. Here, the primary need is to provide 
information that directly relates to particular 
problems, processes, and decisions typically faced 
by managers. Decision-support data of this kind 
typically (and appropriately) resides in pub- 
lished statistical compendia or in microcomputer 
spreadsheet or database files, where aggregate 
information can be easily identified and mani- 
pulated. Indicators of “good practice” of the kind 
described in this Handbook are typically kept in such 
forms. 

Many institutions attempt to generate such 
decision-support information as needed directly 
from operational databases. But this approach 
typically encounters problems, both because the 
data contained in operational databases are not 
always comparable and because the tools needed to 
manipulate the data directly are hard to use. As a 
result, the best approach is generally to create a 
flexible analytical database with the characteristics 
noted in the middle column of Figure 4. Data con- 
tained in this database are periodically extracted 
from operational records systems at defined census 
intervals to ensure consistency. And the file struc- 
tures of this database are designed to allow easy 
manipulation of data elements drawn from many 
disparate sources, as well as calculation of the kinds 
of statistics and ratios required to produce useful 
indicators. The best medium for accomplishing this 
task is a statistical software package (such as SPSS 
or SAS) or a sophisticated database package (such 
as Paradox or Access), resident on either a main- 
frame or a large microcomputer. 

Analytical databases of this kind typically contain 
a set of structured data files designed specifically to 
support academic planning. Each file within the 
database contains a set of pre-defined data elements 
extracted from basic operations systems or from 
unit-level evaluations or assessments in pre-defined 
formats and at designated times. Appropriate extract 
times, of course, will depend upon the nature of 
the data, but for student records data they will 
typically include official census date information 



and end-of-term information. In the case of the 
survey or assessment data prominent in constructing 
indicators of “good practice,” a strong implication is 
that institutions establish a regular cycle for the 
administration of such instruments so that their 
results can be meaningfully compared from year to 
year. Whatever the reporting period used, care 
should be taken that appropriate aggregations across 
different time periods are possible; in the examples 
above, for instance, it should be possible to generate 
meaningful annual information on academic 
activities in relation to annual levels of assessed 
student satisfaction with instruction, even though 
each of these types of data is collected at a different 
point in time. 

Appropriate file structures for such a database 
may vary considerably according to need, but at 
minimum, they include the following components: 

• a< set of term extract files containing basic en- 
rollment and course information, suitable for 
generating a range of historical term statistics and 
indicators. 

• a set of survey or questionnaire files containing 
extracted items of general interest for investigating 
instructional delivery, and drawn from regularly- 
administered student surveys, surveys of faculty, 
or results of end-of-course evaluations. 

• a longitudinal student tracking file assembled 
on a cohort basis and containing the data necessary 
to investigate retention rates, time to degree- 
completion, patterns of coursetaking, and the 
effectiveness of core or developmental course- 
work in terms of later academic performance (see 
Ewell, Parker and Jones 1988). 

• a course/history file assembled on a term basis, 
and containing the data elements needed to 
examine patterns of course attendance by 
discipline and level for particular types of students, 
or patterns of attendance and scheduling. 

Each data element contained in such files requires 
standard definition and should be documented in an 
institutional Data Element Dictionary. Documen- 
tation should contain at minimum a) a conceptual 



35 



definition of the data element in question, b) a 
description of how it is obtained or calculated (e.g., 
through an admissions form), c) which particular 
operational database it is extracted from (noting, if 
relevant, the specific data element name or label 
used in this environment, d) exactly when it is 
collected or extracted, and e) its particular coding or 
categorical structure within the database. Note also 
that many of the data elements included in the 
analytical database are derived from several sources 
or involve additional manipulation of the base data 
elements contained in the operational databases 
from which they were drawn. Where this is the case, 
the specific calculational rules and/or extraction 
procedures used to build these elements should also 
be documented in the Data Element Dictionary. 

Because analytical files of this kind typically 
reside within a standard statistical or database 
software package, manipulating information to 
support flexible data analyses and typical ad hoc 
queries is relatively straightforward. In addition, 
such software packages allow pre-programming a 
range of regular reports and indicators that can 
be run at regular intervals to support ongoing 
institutional and unit-level planning and evaluation 
activities. As an example, such a database might be 
used to generate: 

• a set of overall “good practice” indicators of the 
kinds described in this Handbook 

• a set of program-level performance indicators 
intended to support the institution’s current 
process of academic program review. 

• most of the contents of the institution’s current 
Fact Book consisting of information on enroll- 
ments, academic activity, personnel, fiscal and 
physical condition, and similar items. 

In all three cases, moreover, statistics of these kind 
can be made available not only in print form, but also 
as spreadsheets — allowing academic administrators 
at many levels to extract and manipulate data to meet 
their own local needs. 



Presenting Indicators Results 

Because of the growing salience of indicators to 
both internal and external audiences and the often 
quite different perspectives of different constituents, 
special care is generally needed in communicating 
indicators results. Precisely because they are 
generally indirect, indicators of good practice are 
especially easy to misinterpret if the proper caveats 
are not provided. For external reporting, moreover, 
communicating institutional actions taken in 
response to lower-than-expected indicator results 
can be as important as reporting the results 
themselves. 

A number of issues are associated with presenting 
the results of indicators such as those presented in 
this Handbook, First, results must be placed in their 
proper context in order to avoid significant 
misinterpretations. When additional data that 
provide an interpretational framework for the values 
of specific indicators are provided, misinter- 
pretations of this kind are far easier to avoid. A 
related set of issues involves the eimount and kinds 
of interpretative material to provide. Unaccom- 
panied numbers invite misperception, but too much 
“explanation” in a public report can detract from the 
results themselves, and can appear overly directive 
or defensive. 

Assessment results are especially vulnerable to 
misinterpretation because their values may depend 
far more on intended differences in the particular 
student clienteles served by various institutions or 
programs than on any real differences in unit 
“performance.” Results of “good practice” indi- 
cators are no exception. But how to present such 
differences appropriately can present a major 
challenge for public communication. Past exper- 
ience with reporting results of this kind, however, 
suggests that the following approaches be con- 
sidered: 

• disaggregating results appropriately. One way 

to avoid unfounded “bottom line” comparisons is 

to present results for each academic unit or student 

subpopulation broken down by a number of 
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appropriate factors that may affect performance. 
Here, demographic factors such as gender or 
ethnic background may be useful, as well as 
breakdowns by student major or prior ability. In 
presentations of this kind, institutional or unit 
totals might or might not be provided, depending 
upon the degree to which overall differences in 
performance among units is intended to be 
highlighted. Obviously, use of this method 
depends upon having sufficient numbers of 
students or classes present within each level of 
disaggregation to provide meaningful statistics — a 
factor which may require careful prior planning 
with respect to data collection. 

• providing appropriate standards of compar- 
ison. Absent a visible anchor point forjudging the 
value of any given indicator, different audiences 
will naturally apply different standards. Any 
reporting mechanism should therefore consider 
carefully the kinds of “interpretive” benchmarks 
to be provided. A first alternative here is 
comparisons with established norms of some 
kind. While these will be rare for indicators of 
instructional good practice, they may well be 
available for student behaviors through the results 
of nationally-administered surveys like the CIRP 
or CSEQ — especially for such measures as student 
satisfaction or time-on-task. A second alternative 
is to provide comparisons over time in the form of 
trends. Such reports are especially helpful in 
monitoring progress and in helping to assess the 
effectiveness of any changes in programs of 
policies that may occur. At the same time, trend- 
based presentations allow inevitable statistical 
variations to be shown directly and therefore taken 
into account. Finally, comparisons can be made 
against agreed-upon targets or standards. Based 
on a range of factors, different targets might for 
example be set for different populations or for 



different types of academic units, depending upon 
their specific characteristics or missions. If this 
approach is adopted, care should be taken to ensure 
that such targets are realistic and achievable. 
Achieving such targets, moreover, should not be 
associated with such high stakes that institutions 
or units are tempted to “manage the numbers” 
rather than engaging in real, long-term im- 
provement efforts. 

• using appropriate formats and graphics. A con- 
siderable amount of information can be conveyed 
comparatively and in graphic form, and indicators 
data are usually especially suited to visual 
presentation. Graphics are also effective in some 
contexts and less so in others. In general, graphics 
make good sense for showing trends over time or 
variations in performance for different pop- 
ulations within a range of units or contexts. They 
are less well-suited for presenting single- 
dimensional “bottom-line” data like satisfaction 
levels or performance ratings, where a well-laid- 
out statistical table is generally more effective. 
Regardless of its format, each indicator display 
should also be accompanied by a brief interpretive 
narrative that points out important elements of 
context and that highlights major conclusions. 
Often, narrative of this kind can be incorporated 
directly into the display itself in the form of 
footnotes or titles. 

Used in concert, these principles provide good 
general guidance for presenting the kinds of 
indicators contained in this Handbook. But it is 
important to emphasize that individual contexts and 
experiences may vary, and practitioners are urged to 
pilot various approaches to data presentation on 
different audiences in advance, and to incorporate 
the resulting suggestions into the indicator systems 
that they eventually adopt. 



37 



Chapter 5 



Some Concluding Thoughts 



I ndicators of the kinds described in this 
Handbook have many potential uses. In an 
academic environment increasingly characterized 
by rapid change and multi-faceted modes of 
delivery, they can help academic managers keep 
track of what is happening across a wide range of 
instructional settings. At the same time, such 
information will be more and more required by 
external agencies and by “consumers” of higher 
education faced with complex attendance choices 
and difficult investment decisions. 

But the effective development and use of tools like 
“good practice” indicators requires the adoption of 
a wider management philosophy. Paramount here is 
the commitment to re-direct management infor- 
mation toward core aspects of the educational 
enterprise — to measure what we really value. 
Equally important is the promise to actually use 
information consistently in support of improve- 
ment — a pledge that administrators must make 
continually in every forum available and that they 
must also stand by in the long term, despite the 
implied sacrifice of managerial discretion. Finally, 
good management requires consciousness of the 



limits of all information — especially indicative 
information. While measures of “good practice” 
may point out strengths, weaknesses, and specific 
opportunities for improvement, they should not be 
used on their own to make final decisions. 
Administrators at all levels should resist the 
sometimes compelling temptation to abdicate 
decisional responsibility by creating thresholds and 
formulas that “automatically” determine levels of 
resource investment or particular program and 
institutional priorities. 

Though the context for higher education has 
changed markedly over the past two decades, the 
conceptual principles underlying information- 
based management and decision espoused in 
this Handbook remain in many ways unaltered. 
Embodied in the development of sound indicators of 
instructional good practice and performance — and 
flexibly and appropriately applied — ^these principles 
have new relevance in the turbulent decade of the 
nineties. Distilled to a single phrase for today’s busy 
policymakers and institutional managers, their 
essence can be easily stated: “If it’s important, count 
it; but above all, think before you count.” 
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A Catalog of 

“Good Practice” Indicators 



T his section of the Handbook illustrates a range of sample “good practice” indicators consistent with the 
twelve dimensions presented in Chapter 3. Its intent is to give institutions contemplating the 
development of an indicator system a place to start, but the cajtalog makes no pretence of describing all possible 
indicators of this kind. Indeed, its purpose is well served if institutions make modifications in the sample 
indicators presented to better suit their particular circumstances or to use them as a point of departure for 
developing their own. 

Indicators in this section are organized in terms of the taxonomy presented in Figure 2 in Chapter 3. Each is 
described on a separate page which contains the following information: 

• Indicator. This provides a brief title for the indicator that briefly describes its contents. 

• Description. This entry provides a fuller description of the indicator, its intent and uses, and the manner in 
which is derived or constructed. 

• Associated “Good Practice.” Though already embedded in the reference number, this entry notes for ease 
of reference the particular “good practice” domain associated with the indicator. If more than one such 
domain is potentially so associated, multiple entries are listed with the principal association listed first. 

• Associated Perspective. This repeats the particular “indicator perspective” already embedded in the 
reference number. Again, if more than one perspective is associated with the indicator in question, each is 
listed. 

• Recommended Data Source and/or Calculational Procedure. This entry describes more fully the 
particular manner in which the indicator is collected or constructed. In the case of questionnaire-based 
indicators, for instance, a suggested item text is included. If the indicator is the result of a combination of 
data items such as a percentage or ratio, suggested calculational rules are described for deriving it in a 
consistent fashion. 
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• Reference Number. This is a uniquely-assigned identifier which is used to classify the indicator in question. 
Its first two digits reference the indicator’s principal association with one of the twelve “good practice” 
domains described in Chapter 3; 

a “0” [zero] in this entry notes the fact that the indicator is used to simply describe a curricular context without 
being associated with a particular “good practice.” 

The upper-case letter constituting the third position in the identifying number references one of the three 
additional “cross-cutting” domains described in Chapter 3 according to the following scheme: 

A - institutional context and use of resources, 

B - institutional and curricular requirements, 

C - instructional delivery, and 
D - student experiences. 

A fourth letter in lower-case notes the particular data source used to construct the indicator, as described in 
Chapter 4; here the following scheme: 

a - catalog or syllabus review, 
b - transcripts or student records, 
c - self-reports from faculty and students, and 
d - assessment and samples of student work. 

Finally, within the class of possible indicators described by these combinations, the indicator is given a 
unique, identification number. 

Example: ll.C.b.l 

1 1 = Frequent Faculty-Student Contact 
C = instructional delivery 
b = transcripts or student records 
1 = unique identification number 

In using the catalog, institutions may wish to re-organize entries using the reference numbers provided. For 
instance, if an institution is considering implementing or revising a survey of currently-enrolled students, it 
may wish to sort the catalog on the data-source entry to derive a list of potential candidates for “good practice” 
indicators and their associated question texts as a ready reference. Alternatively, if issues arise about how 
curricular structures are actually being delivered or experienced, it may wish to isolate only those indicators 
associated with curricular coherence and high expectations for consideration — ^regardless of data source. 
Again, the intent is to provide a flexible medium for stimulating good thinking, not to “standardize” on a 
particular format or approach. 
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INDICATOR: Proportion of general education courses or credits in a particular curriculum allocated to 
specific disciplines or areas of study, or to the development of identified prerequisite skills (e.g., writing, math, 
oral communications, etc). 



DESCRIPTION: This is a general indicator of curricular coverage compiled from published catalog or 
syllabus material. Its primary utility is comparative — for instance in benchmarking a given institution’s general 
education offerings against those of peer institutions, or of making comparisons across different program 
requirements within the same institution. 

ASSOCIATED “GOOD PRACTICE”: [Curricular Description] 

ASSOCIATED PERSPECTIVE: Institutional Requirements 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is published catalog material or program descriptions containing the total number of credits 
contained in the curriculum and each of its components. The recommended statistic is a simple proportion 
calculated by dividing each curricular component of interest expressed in credit-unit equivalents, by the credit 
value of the curriculum as a whole. 



REFERENCE NUMBER: O.B.a. 1 
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INDICATOR: The proportion of total credits needed to complete a given program of study that are required 
to be completed in the major discipline. 



DESCRIPTION: This is a measure of overall disciplinary concentration within the designed curriculum, 
intended to reflect how broad or narrow the program is with respect to coverage. Its primary utility is 
comparative — for instance in benchmarking a given institution’s general education offerings against those of 
peer institutions, or of making comparisons across different program requirements within the same institution. 
It can usefully be constructed with denominators using both the total number of credits required for graduation 
and those required to complete a given major. 



ASSOCIATED “GOOD PRACTICE” : [Curricular Description] 

ASSOCIATED PERSPECTIVE: Institutional Requirements 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is published catalog or program descriptions containing listings of the particular courses which 
may be used to satisfy a given requirement. The recommended calculational procedure is construction of a 
simple ratio whose numerator is the number of credits in courses associated with a particular disciplinary listing 
and whose denominator is the total number of credits required to complete the program. An alternative 
“standard” method uses the OERI taxonomy for course classification (Adelman 1990) instead of raw 
department listings to construct the numerator. 



REFERENCE NUMBER: O.B.a.2 
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INDICATOR: Presence of a required demonstration of collegiate writing ability as a condition for graduation 
or program completion. 



DESCRIPTION: This is intended as a very general measure of the degree to which a particular institution, 
program or curriculum sets a standard for mastery of an important cross-cutting skill. Because it is based on 
simple inspection, it does not contain information about the specific skill level set or the degree to which 
students attain it. Similarly, programs may set high standards for writing without having established a specific 
demonstration of competency in this area. 



ASSOCIATED “GOOD PRACTICE”: High Expectations 



ASSOCIATED PERSPECTIVE: Institutional Requirements 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is program or catalog descriptions for the institution or curriculum in question. The indicator is 
a simple flag that notes the presence or absence of the noted curricular feature. Alternatively, an indicator 
might be constructed for comparative purposes that notes the proportion of curricula or programs at the 
institution containing such requirements. 



REFERENCE NUMBER: 1 B.a. 1 
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INDICATOR: Average number of pages of writing [or reading] required by assignments in humanities and 
social science courses. 



DESCRIPTION: This is intended as a direct measure of expected levels of performance as reflected in actual 
course requirements. While it does not reflect the content of such assignments or their quality, it is likely 
reliable for purposes of monitoring levels of expected performance. While a number of data collection methods 
are appropriate, they should probably be used in combination because of the many limitations associated with 
documentary evidence and self-report. 

ASSOCIATED “GOOD PRACTICE”: High Expectations 

Ongoing Practice 

ASSOCIATED PERSPECTIVE: Instructional Delivery 

[Student Experience] 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: Several data 
sources are appropriate for collecting this indicator. The most conceptually straightforward is an analysis of 
course syllabi for the range of courses of interest. Within each course section, assignments can be coded in 
terms of the probable (or explicitly expressed) number of pages involved in completing the assignment, with 
the result averaged across course sections. In the realm of writing, more poweriully, actual student portfolios 
can be constructed using an appropriate previously-identified cross-sectional sample of students. Finally, 
student self-reports on the amount of reading and writing required might be used — either on an overall 
graduating student survey or as items on end-of-course evaluations surveys administered to students enrolled 
in a given class. 

REFERENCE NUMBER: l .C.a.l 
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INDICATOR: Overall distribution of grades granted. 



DESCRIPTION: This is intended as very general contextual measure of the degree to which faculty 
communicate and apply high standards. Extremely high assigned grade levels suggest that high standards are 
not being enforced and that grade inflation may be taking place — although it is important to emphasize that 
small fluctuation in this measure are not meaningful and may indeed communicate wrong messages. 
Nevertheless, the statistic is easily compiled and does directly address one commonly-expressed external 
concern about undergraduate teaching practice. 



ASSOCIATED “GOOD PRACTICE”: High Expectations 



ASSOCIATED PERSPECTIVE: Instructional Delivery 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE. The data source for 
this indicator is the institution’s registration records system for a given term or academic year. The statistic 
itself can be constructed in several ways, but is most commonly compiled as a distribution of the percentage 
of all assigned grades occurring within each grade category. Incompletes or withdrawals may or may not be 
included in the denominator of the percentage calculation, depending upon the intent of the indicator but the 
statistic should clearly report the numbers and/or proportions of such non-graded outcomes present. This 
statistic is also probably most useful when compiled on a unit, program, or class-level basis (e.g., freshmen, 
sophomore, etc). 



REFERENCE NUMBER: 1 C.b. 1 
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INDICATOR: Percentage of students graduating without writing a major research paper during their 
undergraduate career. 



DESCRIPTION : This is intended as general measure of the level of expectations created for all undergraduate 
students. If a significant number of students were never assigned a term research paper of any kind, the 
institution’s general education requirements will likely need attention. A major sustained piece of independent 
writing represents only one of several curricular experiences of this kind that might be tapped as an indicator 
of real faculty expectations. Others might include completion of an experimental or research design, writing 
a critique or evaluation of a piece of writing or reporting, etc. 



ASSOCIATED “GOOD PRACTICE”: High Expectations 

ASSOCIATED PERSPECTIVE : Instructional Delivery 

[Student Experiences] 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: Several data 
sources might be appropriate for this indicator. A complex though direct measure is a transcript analysis for 
an identified cohort of students. Here the calculation would demand previous coding of courses to determine 
whether the requirement of interest is in fact present (e.g., if a research paper of some kind was assigned), then 
searching each student’s transcript record to determine if at least one such course was present. Such coding, 
of course, may require arbitrary judgments to be made about whether or not a particular requirement is in fact 
present. Once these are identified, the statistic is the number of students without such an occurrence divided 
by the total number of graduating students in the cohort. Alternatively, a self-report might be used; a suggested 
question text is, “In any of your classes as an undergraduate, did you have to prepare a paper that investigated 
in depth a topic that you chose yourself and that required substantial library or original research?” 

REFERENCE NUMBER: l .C.c.l 
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INDICATOR: Percentage of students completing their freshman year without initiating a library search [or 
checking a book out of the library]. 



DESCRIPTION: This is intended as an indirect behavioral measure of the degree to which faculty create an 
expectation of library use through the assignments that they require. Because of differences among disciplines, 
it should only be used as an overall measure and, in the case of actually checking material out, should be treated 
as a “bounding” indicator, with extremely low values suggesting that low expectations or study habits are 
present. 



ASSOCIATED “GOOD PRACTICE”: High Expectations 

Early Years of Study 



ASSOCIATED PERSPECTIVE: Student Experiences 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE. The data source for 
this indicator is library records and requires the individual identification of student activity at the library. For 
materials actually checked out, these records are typically present on a student-by-student basis although 
because of the ways they are archived, compiling them for an annual period may prove challenging. For 
log-ons, searches or other transactions, such identification may present a larger challenge or not be possible at 
all. Alternatively, a self-report from graduating students might be used; a suggested question text is, “During 
your undergraduate years, in addition to studying and reserve reading, about how many times per week did 
you visit the library to look something up or to pursue a research question?” 



REFERENCE NUMBER: 1 D.b. 1 
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INDICATOR: Percentage of students reporting not being significantly challenged by class material and 
assignments. 



DESCRIPTION: This is intended as a measure of students’ own perceptions about the levels of expectation 
to which they are held accountable. Though a self-report, it constitutes a valuable indicator in its own right. 
If students feel unchallenged, the actual levels of achievement expected by faculty will likely bear looking at 
even though these may be high. Conversely, variations in perceived levels of challenge render this indicator 
inappropriate for comparisons among different types of institutions (though they may be useful to compare the 
experiences of different types of students). 



ASSOCIATED “GOOD PRACTICE” : High Expectations 



ASSOCIATED PERSPECTIVE: Student Experience 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is additional item(s) included in an end-of-course evaluation survey administered to all enrolled 
students. A suggested question text is, “About how difficult or challenging did you find the material presented 
in this class? Was it presented at about the right level for you?” Alternatively, a graduation student survey 
item might be used, requesting respondents to rate on a five-point scale the overall difficulty or challenge 
presented by courses in their overall undergraduate career. In this case, courses in the major vs. general 
education or by year of study might usefully be distinguished. 



REFERENCE NUMBER: 1 D.c. 1 
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INDICATOR: The number of courses among which students can choose in order to meet a given curricular 
requirement. 

DESCRIPTION; This indicator constitutes a rough measure of the degree of undirected choice students have 
in fulfilling a given curricular requirement, used for purposes of general description. Because it is particularly 
suited to examining requirements in distribution-based curricular schemes, it is especially applicable for 
examining general education programs. 

ASSOCIATED “GOOD PRACTICE”: Coherence in Learning 

ASSOCIATED PERSPECTIVE: Institutional Requirements 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is published catalog or program descriptions containing listings of the particular courses which 
may be used to satisfy a given requirement. While an absolute count by field or area can be useful, the indicator 
is most beneficially calculated by dividing the number of courses available among which students may choose 
to fulfill a given requirement by the number of courses needed from this list to complete the requirement. 

REFERENCE NUMBER: 2.B.a.l 
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INDICATOR: The proportion of the courses needed to complete a particular program of study that require 
prerequisites. 



DESCRIPTION: This is a general measure of the sequence and structure of a given curriculum and can be 
constructed in a number of ways. Its primary utility is comparative — for instance in benchmarking a given 
institution’s general education offerings against those of peer institutions, or of making comparisons across 
different program requirements within the same institution. Its essential logic is to determine how many of 
the courses required for completion themselves have prerequisites, and how many pre-requisites they have, in 
order to provide a measure of the “linearity” of the program. Programs that are high on this measure are 
elaborately structured. In addition, transferring into such programs will probably entail additional time to 
graduation. Programs low on this measure are relatively loose in structure, and it is unlikely that many intended 
connections among courses are intended in their design. 



ASSOCIATED “GOOD PRACTICE”: Coherence in Learning 

ASSOCIATED PERSPECTIVE: Institutional Requirements 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is published catalog or program descriptions containing listings of the particular courses which 
may be used to satisfy a given requirement. The calculational procedure recommended is first to sort courses 
required for completion in a given program according to the number of pre-requisites that they require, then 
create a proportion for each “prerequisite group” so established. For instance, an institution might choose to 
construct a separate statistic for the proportion of total courses required that consist respectively of, a) courses 
that have no pre-requisites, b) courses that have one prerequisite, c) courses that have two prerequisites and, 
d) courses that have three or more pre-requisites. 



REFERENCE NUMBER: 2.B.a.2 
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INDICATOR: Proportion of first -year [or 100-level] courses in writing or mathematics containing 
assignments explicitly linked to other discipline-specific classes. 



DESCRIPTION: The intent of this indicator is to reflect conscious course designs and/or teaching practices 
that explicitly encourage students to make linkages across individual classes. These might include deliberate 
course designs in which assignments in disciplinary courses are used as settings for skills development (e.g., 
at least one paper in a composition course is also a paper for another course in which the student is concurrently 
enrolled) or assignments in skills courses that are consciously structured to reflect the disciplinary contexts in 
which students will later use the skill (e.g., calculus problems that are constructed around engineering or physics 
settings). 



ASSOCIATED “GOOD PRACTICE:” Coherence in Learning 



ASSOCIATED PERSPECTIVE: Instructional Delivery 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is assembled syllabus material which must be coded manually to reflect the presence or absence 
of such features. Typically, these will be present in the assignments given to students and it may be appropriate 
to set a minimum criterion for inclusion based on the number of such assignments or the proportion of a 
student’s grade that such exercises represent. Calculation of the statistic is then based on a simple ratio between 
courses containing such features and the total number of courses in the domain of interest. Alternatively, a 
survey of instructional staff might be used to collect this indicator by requesting respondents to identify the 
frequency with which they construct assignments that explicitly reference other course or disciplinary contexts. 



REFERENCE NUMBER: 2.C.a. 1 
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INDICATOR: Percentage of students reporting that they did [or were required to] make explicit connections 
between what they learned in a particular course and another course that they were taking. 



DESCRIPTION: This is intended as a direct measure of student experiences in linking knowledge and skills 
across course contexts — an important element in coherence. As noted, it might be constructed either to reflect 
instructional delivery or actual experiences whether or not such connections were explicitly required. As a 
self-report, it is inherently limited as students will vary in their perceptions of what constitutes such a 
“connection,” but the indicator is straightforward and easily included in a typical end-of-course questionnaire. 



ASSOCIATED “GOOD PRACTICE”: Coherence in Learning 

ASSOCIATED PERSPECTIVE: Student Experience 

Instructional Delivery 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is an additional item included in an end-of-course evaluation survey administered to all enrolled 
students. A suggested question text is, “About how often in this class over the course of the term did you [or 
were you required to] make a specific connection between something that you learned in this class with an 
assignment or topic in another class that you were taking.” 



REFERENCE NUMBER: 2.D.C.1 
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INDICATOR: Proportion of curricula or programs requiring a capstone course, senior project, or other 
end-of-program synthesizing experience. 



DESCRIPTION: The intent of this indicator is to provide an overall description of the extent to which 
programs contain curricular features consistent with the principle of synthesizing experience. Examples of 
activities consistent with this “good practice” are many and include capstone courses, senior theses or 
independent projects, comprehensive examinations, and internships or practicum settings. The principal 
requirement is that such experiences actively require the student to assemble and use knowledge and skill 
elements developed throughout the program in a manner that demonstrates that he or she can deploy them 
appropriately and flexibly, and understands the relationships among them. 

ASSOCIATED “GOOD PRACTICE”: Synthesizing Experiences 



ASSOCIATED PERSPECTIVE: Institutional Requirements 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is catalog and syllabus material for the institution’s programs and curricula. The indicator itself 
requires a simple flagging of each program or curriculum that requires such an experience as a condition of 
graduation. 



REFERENCE NUMBER: 3.B.a.l 
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INDICATOR: Percentage of graduating students indicating that they were required to actively synthesize 
knowledge drawn from many courses and/or disciplines. 



DESCRIPTION: The intent of this indicator is to directly measure completing students’ perceptions of the 
degree to which they were asked to put together knowledge and skills in the assignments which they were 
given in their final year. As a self-report, it is limited because students may vary in the degree to which they 
recognize that such “synthesizing experiences” in fact occurred. In developing this indicator, moreover, 
distinctions might be made about whether such synthesizing experiences were present throughout the student’s 
experience and in the major field. 



ASSOCIATED “GOOD PRACTICE”: Synthesizing Experiences 



ASSOCIATED PERSPECTIVE: Student Experience 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is an item included on a graduating student questionnaire. A suggested question text is, “During 
your final year of study, to what extent did the assignments you were given require you to put together what 
you learned in individual courses taken throughout your program to be successful?” A five-point scale is 
recommended. 



REFERENCE NUMBER: 3.D.C. 1 
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INDICATOR: Number of internships, practica, or other practice-oriented courses offered per student. 



DESCRIPTION: This is intended to be a measure of the opportunity students have to engage in coursework 
explicitly designed to foster active or independent modes of learning and fuse education and experience. Tlie 
types of sections noted are notable for their inclusion of such learning modalities and are probably easily 
identifiable within the institution’s curriculum, but do not represent the total universe of such courses. Because 
they involve self-directed learning, independent study sections might also be included in the calculation of 
such a ratio, or might be used to construct a separate indicator. If the institution can identify further courses 
that can be determined to contain applied or active learning features such as self-paced modules, independent 
study, extensive teamwork, or substantial use by faculty of applied learning techniques in the classroom, these 
might also be included. As in other such measures, however, the indicator reflects only the opportunity to 
engage in active learning; it does not directly measure whether or the degree to which this kind of learning 
actually takes place. 

ASSOCIATED “GOOD PRACTICE”: Integrating Education and Experience 

Active Learning 

ASSOCIATED PERSPECTIVE: Institutional Requirements 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source 
for this indicator is the institution’s registration records system for a given term or academic year. Calculation 
of the statistic requires courses containing learning components of interest to be readily identifiable in 
registration records — either because they have distinctive numbers or a special flag. Once identified, 
calculation of the indicator is a straightforward ratio between the number of such courses and total student 
enrollment calculated on a headcount or FTE basis. 



REFERENCE NUMBER: 4.B.b.l 
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INDICATOR: Percentage of graduating students engaging in at least one internship, practicum, independent 
study, or similar self-directed or practice-oriented course during an undergraduate career. 



DESCRIPTION: This indicator is intended to be a measure of the extent of total student exposure to courses 
that involve applied or active learning experiences. The types of sections noted are notable for their inclusion 
of such learning modalities and are probably easily identifiable within the institution’s curriculum, but do not 
represent the total universe of such courses. If the institution can identify further courses that can be determined 
to contain extensive similar features such as self-paced modules, independent study, extensive teamwork, or 
substantial use by faculty of active applied learning techniques in the classroom, these might also be included. 
As in other such measures, participation in such courses does not in itself guarantee that such learning is taking 
place. 



ASSOCIATED “GOOD PRACTICE”: Integrating Education and Experience 

Active Learning 



ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The primary data 
source for this indicator is the institution’s records system for an identified cohort of graduating students. Its 
calculation requires courses containing practice-oriented components to be readily identifiable in registration 
records — either because they have distinctive numbers or a special flag. Once these are identified, the statistic 
is calculated by searching the transcript record of each completing student and determining if course records 
are present that are appropriately identified. If one or more such courses are indeed present, the student is 
counted in the numerator of the statistic with the denominator being the total number of graduating students. 
Alternatively, the indicator can be collected through a graduating student questionnaire. A suggested question 
text is, “In the course of your undergraduate career, did you take any courses that required you to apply what 
you learned in real settings such as independent studies, practica, or internships?” 

REFERENCE NUMBER: 4.D.b. I 
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INDICATOR: Percent of students involved in faculty research. 

DESCRIPTION: This is intended as a direct measure of an effective practice for synthesizing education and 
experience, obtained by self-report from graduating seniors. This indicator reflects only engagement or 
frequency of such activity but additional indicators might be developed to reflect student’s perceptions of the 
quality of such experiences and any associated learning gains. Furthermore, parallel indicators might be 
developed based on faculty reports of involvement with students in research activities. 



ASSOCIATED “GOOD PRACTICE”: Integrating Education and Experience 

Student/Faculty Contact 

ASSOCIATED PERSPECTIVE: Student Experiences 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The primary data 
source for this indicator is appropriate items on a graduating student survey. A suggested question text is, “Did 
you ever assist a faculty member on a project relating to his or her research?” Alternative formats might tap 
the frequency with which such activities occurred by year. For faculty, a suggested text for inclusion in a 
periodic faculty survey is, “How many undergraduate students directly participated in your research this year?” 

REFERENCE NUMBER: 4.D.C.1 



59 



INDICATOR: Percentage of courses requiring students to engage in independent research papers, projects, 
presentations, or similar exercises. 



DESCRIPTION: The intent of this indicator is to examine the overall proportion of courses in a particular 
array that include designed practices consistent with active learning. Because its source is published syllabus 
material, the statistic may not directly reflect the presence of such practices for two reasons: they may be present 
and not documented in the syllabus, or they may be claimed in the syllabus and not actually practiced. Because 
of these drawbacks, the statistic should be interpreted with caution. The statistic is also most useful in 
comparing different subsets of courses (e.g., across departments, by level, etc.). 

ASSOCIATED “GOOD PRACTICE”: Active Learning 

ASSOCIATED PERSPECTIVE: Institutional Requirements 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is assembled syllabus material which must be coded manually to reflect the presence of 
appropriate opportunities or practices. Typically, these will be present in the assignments given to students, 
and it may be appropriate to set a minimum criterion for inclusion based on the number or such practices or 
the proportion of a student’s grade that such exercises represent. Calculation of the statistic then is based on 
a simple ratio between courses containing such practices and the total number of courses. Because of the 
difficulty of accomplishing such coding, the indicator might be calculated only on the basis of a random sample 
of classes, or for classes that meet particular criteria (e.g., 1 00-level, offered for general education credit, etc.). 

REFERENCE NUMBER: S.B.a.l 
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INDICATOR: Percentage of courses explicitly requiring students to use the library as a research resource. 

DESCRIPTION: The intent of this indicator is to examine the overall proportion of courses in a particular 
array that include library use, which is consistent with active learning. Because its source is published syllabus 
material, the statistic may not directly reflect the presence of library-related assignments for two reasons: they 
may be present and not documented in the syllabus, or they may be claimed in the syllabus and not actually 
practiced. Because of these drawbacks, the statistic should be interpreted with caution. In addition, it can be 
cross-checked with student self-reports about the amount of library work assigned and their own use of the 
library. 

ASSOCIATED “GOOD PRACTICE”: Active Learning 



ASSOCIATED PERSPECTIVE: Instructional Delivery 

[Student Experience] 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is collected syllabus material which must be coded manually to reflect the presence of appropriate 
library or other research-related independent projects. Typically, these will be present in the assignments given 
to students, and it may be appropriate to set a minimum criterion for inclusion based on the number or such 
practices or the proportion of a student’s grade that such exercises represent. Calculation of the statistic then 
is based on a simple ratio between courses containing such practices and the total number of courses. Because 
of the difficulty of accomplishing such coding, the indicator might be calculated only on the basis of a random 
sample of classes, or for classes that meet particular criteria (e.g., 100-level, offered for general education 
credit, etc.). If a student self-report is used in addition, the recommended data source is additional items 
included in an end-of-course evaluation survey administered to all enrolled students. A suggested question 
text is, “With the exception of studying and doing reserve reading, about how many times did you visit the 
library this term to find out something that you needed to know for this class?” 



REFERENCE NUMBER: S.C.a.l 
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INDICATOR: Percentage of faculty reporting giving students credit for active participation in classroom 
discussions. 



DESCRIPTION: This is intended to be an indicator of the degree to which faculty formally encourage 
participation in class discussions in evaluating student performance. Because it is presented in the form of a 
self-report, it should be treated with caution as actual practice may differ from what is reported. As a result, 
it might be cross- validated by a parallel statistic collected from students in an end-of-course questionnaire. 



ASSOCIATED “GOOD PRACTICE”: Active Learning 

ASSOCIATED PERSPECTIVE: Instructional Delivery 

[Student Experience] 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is a survey administered periodically to instructional staff A suggested question text is, “In your 
classes this year, did you give any grade credit to students for active participation in class?” If the alternative 
of an item on an end-of-course evaluation questionnaire for students is used, a suggested question text is, “Did 
the instructor in this class require you to participate actively in class discussions and was the degree or quality 
of your participation counted as part of your grade?” 



REFERENCE NUMBER: 5.C.C.1 
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INDICATOR: Average number of items checked out of the library by undergraduates per undergraduate 
student. 



DESCRIPTION: This is intended to provide an indirect indicator of the degree to which students are engaged 
in independent reading, whether related or not to course assignments. Because it reflects only a single source 
of reading materials, it is inherently limited in scope — excluding student purchase of books and use of other 
library and information resources. Nevertheless, it may provide a reasonable benchmark of an important 
student activity. 



ASSOCIATED “GOOD PRACTICE” : Active Learning 

ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is library records, and requires materials checked out to be individually identifiable as to whether 
these students are enrolled as undergraduates. If circulation can be so identified, the calculation of the statistic 
is a simple average of items checked out divided by the total number of enrolled undergraduates calculated on 
a headcount or FTE basis. Alternatively, appropriate ranges of the number of items checked out could be 
constructed — e.g., none, less than five, etc. In addition, if library records are kept by student number, the 
statistic can be constructed for different types of students — e.g., by program, level, or demographic 
characteristic. 



REFERENCE NUMBER: 5.D.b. 1 
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INDICATOR: Number of library computer searches or database log-ons initiated by undergraduates per 
undergraduate student. 

DESCRIPTION: The intent of this indicator is to indirectly track active student use of the institution’s 
information resources. It is indicative only, as no information is typically available about the actual content 
of such log-ons or searches. Construction of the indicator requires that such activity be individually identifiable 
by student when it occurs — a situation which may be the case for certain kinds of activities (e.g., computer 
network access) and not for others (on-line public catalog access in the library). On the other hand, the 
technology of such media may allow additional relevant data to be captured — for example the duration, 
complexity, and sophistication of searches initiated or executed. 

ASSOCIATED “GOOD PRACTICE” : Active Learning 

ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is library and other computer database records of access and use. If each access can be 
individually identified by student through an account or password, the statistic is calculated by determining 
the number of such instances per student, then dividing by the total number of undergraduate students counted 
on a headcount or FTE basis. Alternatively, a student self-report can be used, administered on an instrument 
such as a graduating student survey. 



REFERENCE NUMBER: 



5.D.b.2 
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INDICATOR: Percentage of students in a given class section reporting having asked at least two or three 
questions in class during the term. 

DESCRIPTION: This is intended to be a direct measure of student classroom participation, which research 
suggests is closely related to active learning. As a self-reported measure, it should be treated with some caution 
but the statistic provides a relatively robust and easy-to-collect indicator of one of the most common student 
behaviors associated with active learning. The number of questions asked is arbitrary, and may be set 
appropriately at any level that discriminates effectively among students to reflect varying levels of actual 
participation. 

ASSOCIATED “GOOD PRACTICE” : Active Learning 

ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is an additional item or items included in an end-of-course evaluation survey administered to all 
students enrolled in the course. A suggested question text is, “About how many times did you actively raise 
a question during class this term?” 

REFERENCE NUMBER: 5.D.C. 1 
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INDICATOR: Average number of pages of out-of-class wanting assigned in courses in the humanities and 
social sciences. 



DESCRIPTION: The intent of this indicator is to provide an overall estimate of the amount of writing 
occurring in appropriate courses — a direct indicator of ongoing practice of a key academic skill. In courses of 
this kind, considerable writing is also consistent with active learning. Because the source of the indicator is 
published syllabus material, the statistic may not directly reflect the actual amount of writing required for two 
reasons: additional assignments may be present and not documented in the syllabus, or they may be claimed 
in the syllabus and not actually practiced. Because of these drawbacks, the statistic should be interpreted with 
caution. 



ASSOCIATED “GOOD PRACTICE”: Ongoing Practice 

Active Learning 
High Expectations 

ASSOCIATED PERSPECTIVE: Instructional Delivery 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is collected syllabus material which must be coded manually to determine the amount of writing 
required. Typically, this information will be present in the assignments given to students. Calculation of the 
statistic then is based on a simple average of the approximate number of pages of writing required in target 
courses, divided by the number of courses. Because of the difficulty of accomplishing such coding, the 
indicator might be calculated only on the basis of a random sample of classes, or for classes that meet particular 
criteria (e.g., 100-level, offered for general education credit, etc.). 



REFERENCE NUMBER: 6.C.a. 1 
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INDICATOR: Percentage of students reporting that instructors in a given class held review or study sessions 
in addition to regular class time. 



DESCRIPTION: This is intended as an indicator of the commitment of faculty to furthering student practice 
of learned skills and more general time on task. It might be usefully supplemented by an additional self-reported 
indicator on whether and how often students in fact participated in such activities. 



ASSOCIATED “GOOD PRACTICE”: Ongoing Practice 

Time on Task 



ASSOCIATED PERSPECTIVE: Instructional Delivery 

Student Experience 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is an additional item or items included in an end-of-course evaluation survey administered to all 
students enrolled. A suggested question text is, “Did your instructor in this class hold extra study or review 
sessions in addition to regular class meetings, and how often did you participate in these sessions?” 



REFERENCE NUMBER: 6.C.c. 1 
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INDICATOR: Average number of hours per week reported as having been spent reading on one’s own by 
graduating seniors. 

DESCRIPTION: This is intended as an indirect measure of student out-of-class activities consistent with 
ongoing practice and academic time on task. While the reading that it addresses may or may not reflect 
academic content, considerable research suggests that reading itself — regardless of content — is related to 
overall academic achievement. As a further refinement, different types of reading might be distinguished — for 
instance by content (e.g., reading directly or indirectly associated with specific classwork vs. personal interest) 
or by type of material (e.g., books vs. periodicals and magazines). At the same time, research suggests 
monitoring an important alternative to independent reading: self-reported hours of watching television are 
generally a strong negative predictor of academic achievement presumably because of its impact on time on 
task (Astin 1992). As a self-reported retrospective measure, all such items are to be treated with caution. 

ASSOCIATED “GOOD PRACTICE”: Ongoing Practice 

Time on Task 



ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is items included on a graduating student survey. A suggested question text is, “As an 
undergraduate, about how many hours per week did you spend reading on your own beyond what was actually 
assigned in your classes?” 

REFERENCE NUMBER: 6.D.C.2 
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INDICATOR: Average number of graded assignments or exercises given per course. 



DESCRIPTION: The intent of this indicator is to provide an overall estimate of the distinct number of 
assignments given to students in a given course. Larger numbers of assignments imply more frequent feedback 
given to students, as well as providing more opportunity to practice learned skills. Because the source of the 
indicator is published syllabus material, the statistic may not directly reflect the actual number of assignments 
for two reasons: they may be present and not documented in the syllabus, or they may be claimed in the syllabus 
and not actually practiced. Because of these drawbacks, the statistic should be interpreted with caution. 

ASSOCIATED “GOOD PRACTICE”: Assessment and Feedback 

Ongoing Practice 

ASSOCIATED PERSPECTIVE: Instructional Delivery 

RECOMMENDED DATA SOURCE AND/OR CALCULATION AL PROCEDURE: The data source for 
this indicator is collected syllabus material which must be coded manually to determine the number of distinct 
assignments given to students. Calculation of the statistic then is based on a simple average of the approximate 
number of assignments required in target courses, divided by the total number of courses. Because of the 
difficulty of accomplishing such coding, the indicator might be calculated only on the basis of a random sample 
of classes, or for classes that meet particular criteria (e.g., 1 00-level, offered for general education credit, etc.). 
Alternatively, it might be calculated differently for distinct types of courses — for example counting the distinct 
number of graded writing assignments per course in freshman courses in the humanities or social sciences, or 
the number of graded tests and quizzes in freshman science and math courses. 

REFERENCE NUMBER: 7.C.a. 1 
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INDICATOR: Percentage of courses requiring a graded assignment within the first two weeks of the term. 



DESCRIPTION: The intent of this indicator is to provide an indicator of faculty practices consistent with 
providing prompt feedback to students. Early quizzes or other exercises enable faculty to provide timely 
assessments of student performance needed for intervention or self-correction. Because the source of the 
indicator is published syllabus material, the statistic may not directly reflect the presence of such assignments 
for two reasons: they may be present and not documented in the syllabus, or they may be claimed in the syllabus 
and not actually practiced. Because of these drawbacks, the statistic should be interpreted with caution. 



ASSOCIATED “GOOD PRACTICE”: Assessment and Feedback 



ASSOCIATED PERSPECTIVE: Instructional Delivery 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is collected syllabus material which must be coded manually to determine the number of such 
assignments given to students. Calculation of the statistic then is based on a simple average of the approximate 
number of early-graded assignments, divided by the number of courses. Because of the difficulty of 
accomplishing such coding, the indicator might be calculated only on the basis of a random sample of classes, 
or for classes that meet particular criteria (e.g., 100-level, offered for general education credit, etc.). 
Alternatively, a faculty self-report might be used; in this case the relevant indicator would be the percentage 
of faculty reporting such practices. 



REFERENCE NUMBER: 7.C.a.2 
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INDICATOR: Percentage of courses allowing or requiring multiple drafts, re-writes, or re-submission of 
student work. 

DESCRIPTION: The intent of this indicator is to examine the overall proportion of appropriate courses that 
allow or encourage writing practices consistent with providing extensive feedback to students on their 
performance. Because its source is published syllabus material, the statistic may not directly reflect the 
presence of such practices for two reasons: they may be present and not documented in the syllabus, or they 
may be claimed in the syllabus and not actually practiced. Because of these drawbacks, the statistic should be 
interpreted with caution. In addition, it can be cross-checked with student self-reports about the presence and 
use of such practices. 



ASSOCIATED ‘‘GOOD PRACTICE”: Assessment and Feedback 

Ongoing Practice 



ASSOCIATED PERSPECTIVE: Instructional Delivery 

[Student Experience] 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is collected syllabus material which must be coded manually to reflect the presence of assignments 
requiring or allowing re-submission or revision. Calculation of the statistic then is based on a simple ratio 
between courses containing such practices and the total number of courses. Because of the difficulty of 
accomplishing such coding, the indicator might be calculated only on the basis of a random sample of classes, 
or for classes diat meet particular criteria (e.g., 1 00- level, offered for general education credit, etc.). If a student 
self-report is used in addition, the recommended data source is additional items included in an end-of-course 
evaluation survey administered to all enrolled students. A suggested question text is, “Were you encouraged 
or required by the instructor in this course to revise or resubmit your work after he or she provided you with 
comments on how it might be improved?” 



REFERENCE NUMBER: 7.C.a.3 
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INDICATOR: Percentage of students reporting that they generally receive graded assignments back from 
instructors within one week. 



DESCRIPTION: This is intended as a direct measure of the promptness with which faculty provide students 
with feedback on their formally- submitted work. The time limit is arbitrary, and might alternatively be 
expressed in terms of elapsed class meeting periods. While based on a student self-report, the statistic provides 
what is probably the single best measure of overall commitment to the principal of frequent feedback. 



ASSOCIATED ‘‘GOOD PRACTICE”: Assessment and Feedback 



ASSOCIATED PERSPECTIVE: Student Experience 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is an additional item included on an end-of-course evaluation survey administered to all students 
enrolled in the course. A suggested question text is, “About how long did it generally take to get graded 
assignments back from your instructor in this class?” 



REFERENCE NUMBER: 7.D.C. 1 
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INDICATOR: Percentage of students reporting that the instructor systematically reviewed tests or assignments 
in class after papers were returned. 



DESCRIPTION; This is intended as a direct measure of the commitment of faculty to the principal of 
providing one dimension of feedback — collective “coaching” of students on a common exercise. Although 
self-reported, students can be expected to provide information of reasonable accuracy. Additional items might 
be designed in parallel to examine student satisfaction with or their perception of the helpfulness of such in-class 
review procedures. At the same time, items reflecting such practices might be included on a faculty survey to 
determine the fi'equency with which faculty claim to engage in them. 

ASSOCIATED “GOOD PRACTICE” : Assessment and Feedback 



ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE. The data source for 
this indicator is additional item(s) included in an end-of-course evaluation survey administered to all students 
enrolled in the course. A suggested question text is, “About how often did the instructor in this class go over 
assignments or tests afterwards so that you could see and correct your mistakes?” 



REFERENCE NUMBER: 7.D.C.2 
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INDICATOR: Percentage of students reporting that the instructor provided frequent specific conunents on 
their own performances. 



DESCRIPTION: This is intended as a direct measure of the commitment of faculty to the principal of 
providing individualized feedback directly to students. Although self-reported, students can be expected to 
provide information of reasonable accuracy. Additional items might be designed in parallel to examine student 
satisfaction with or their perception of the helpfulness of such comments. At the same time, items reflecting 
such practices might be included on a faculty survey to determine the frequency with which faculty claim to 
engage in them. 



ASSOCIATED “GOOD PRACTICE”: Assessment and Feedback 



ASSOCIATED PERSPECTIVE: Student Experience 



RECOMMENDED DATA SOURCE AND/OR CALCULATION AL PROCEDURE: The data source for 
this indicator is additional item(s) included in an end-of-course evaluation survey administered to all students 
enrolled in the course. A suggested question text is, “About how often did the instructor in this class provide 
you with specific comments on your performance intended to help you improve?” 



REFERENCE NUMBER: 7.D.C.3 
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INDICATOR: Percentage of graduating students reporting that they received frequent comments and 
evaluations during their undergraduate careers, and that such feedback helped them to improve their 
performance. 

DESCRIPTION: This is intended as an overall indication of the degree to which frequent feedback is practiced 
across the curriculum as a whole. Because it is a self-reported measure provided retrospectively, it may not 
accurately reflect the early years of study and will tend to blur distinctions among different types of courses. 
For this reason, additional indicators might be developed that break down such evaluations by types of courses 
(e.g., major field vs. general education, by level, or by discipline). Separate items might also be developed 
tapping the perceived frequency and helpfulness of instructor comments. 



ASSOCIATED “GOOD PRACTICE”. Assessment and Feedback 



ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is a periodic survey administered to graduating students. A suggested question text for examining 
frequency might be, “In your classes as an undergraduate, about how often did you receive individual comments 
on your work from your instructors?” A suggested question text for examining feedback quality might be, “In 
your classes as an undergraduate, how would you rate the overall helpfulness of the comments that your 
instructors gave you?” In the latter case, a five-point rating scale is recommended. 



REFERENCE NUMBER: 7.D.C.4 
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INDICATOR: Percent of courses [by department, program] incorporating team projects or similar group 
learning experiences. 



DESCRIPTION: This is intended as a direct measure of course designs containing collaborative features, as 
reflected in associated syllabi. Many examples of such features are possible, but their defining characteristics 
are that students are expected to work interactively and that results will be assessed for the team as a whole 
(although individual contributions may be assessed as well). Because the indicator is based on documentary 
records, it may contain two kinds of errors. First, plans for such efforts may not occur as described in syllabi. 
More likely, faculty may include such course features in their actual course delivery, but neglect to document 
them in their syllabi. As a result, this indicator should be used cautiously, and in conjunction with student 
reports on such curricular features. 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is a content analysis of course syllabi for the range of courses of interest. Courses should be coded 
for the number of instances of teamwork or group projects explicitly mentioned in the assignments contained 
in the syllabus. Alternatively, an indicator might be developed that reflects the proportion of each student’s 
graded work that occurs in a team setting, and this proportion averaged across the range of courses of interest. 

REFERENCE NUMBER: S.C.a.l 



ASSOCIATED “GOOD PRACTICE”: Collaborative Learning 



ASSOCIATED PERSPECTIVE: 



Instructional Delivery 



76 




INDICATOR: Percent of faculty reporting efforts to create group projects or “learning communities” in their 
classes. 

DESCRIPTION: This is intended to create opportunities for collaborative learning and cooperation among 
students in their classes. As a self-reported measure, it should be treated with some caution, and should be 
cross-checked with student responses and/or other sources to confirm its presence. Also, such opportunities 
may be more difficult to create in certain kinds of classes, and this should be taken into account in interpreting 
the resulting statistics. 

ASSOCIATED “GOOD PRACTICE”: Collaborative Learning 



ASSOCIATED PERSPECTIVE: Instructional Delivery 

RECOMMENDED DATA SOURCE AND/OR C ALCULATION AL PROCEDURE: The data source for 
this indicator is a periodic survey of instructional staff. A suggested question text is, “In your classes this [year, 
term] did you make any attempts to establish student study groups, ‘learning communities,’ or other formal 
opportunities for students to learn from one another?” 



REFERENCE NUMBER: 8.C.C.1 
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INDICATOR: Percent of faculty using non-competitive grading criteria. 



DESCRIPTION: This is intended as a direct measure of faculty attempts to apply cooperative practices to 
their assessment of students in the form of grades awarded and feedback provided. More specifically, the 
indicator rests on the distinction between traditional normative grading practices in which students are ranked 
against one another to produce a single “normal” distribution for the class, and “criterion-based” alternatives 
in which explicit fixed performance standards are established and students assessed against them individually. 
In the latter case, students are not ranked against one another and all (or none) of the students in a given class 
may achieve a given level. Because students are not competing against one another for a grade, the latter 
approach is generally strongly related to collaboration among students. As a self-reported measure, it should 
be treated with caution, but it is probably best gathered in this manner as students will likely be less aware of 
the particular philosophy of grading applied. 



ASSOCIATED “GOOD PRACTICE”: Collaborative Learning 

Assessment and Feedback 



ASSOCIATED PERSPECTIVE: Instructional Delivery 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE. The data source for 
this indicator is a periodic survey of instructional staff. A suggested question text is, “Which of the following 
best describes the way you assigned grades in your classes this year, a) ranking student performance in the 
class to achieve an approximately ’’normal” distribution of assigned grades, or b) assigning grades associated 
with specific mastery or competency levels without ranking students?” 



REFERENCE NUMBER: 8.C.C.2 
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INDICATOR: Percent of students reporting participation in team projects or group study. 



DESCRIPTION: This is intended as a direct measure of actual instances of student collaboration for a single 
course, as reported by students. Examples such cooperative activities may be many and include 
deliberately-designed opportunities such as team projects or “learning communities,” and more informal 
student-initiated instances of cooperation such as study groups. Because the indicator is based on a self-report 
on a matter which has no stakes for students, it is likely to be a reliable measure of at least the frequency with 
which such practices occur. It will not likely be able to assess the quality of such occurrences or their 
contribution to learning. 



ASSOCIATED “GOOD PRACTICE” : Collaborative Learning 

ASSOCIATED PERSPECTIVE: Student Experiences 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is an additional item or items included in an end-of-course evaluation survey administered to all 
students enrolled. A suggested question text for formal course opportunities is, “Were any group or team 
projects assigned as a part of this course, and what was your level of participation in them?” A suggested 
question text for informal collaborative activities is, “Did you ever [or how many times did you] get together 
regularly with other students to study or prepare for this class?” Similar items might be constructed for 
inclusion in a survey of graduates; here a suggested question text is, “In the classes that you took as an 
undergraduate, about how frequently did you study or prepare assignments in collaboration with other 
students?” 



REFERENCE NUMBER: S.D.c. 1 
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INDICATOR: Percent of graduating seniors reporting frequent informal conversations with academic content 
with fellow students. 



DESCRIPTION: This indicator is intended to provide an overall measure of the frequency of conversations 
about academic topics occurring in the student population. Though very general, it can provide a useful 
benchmark for tracking the experiences of particular student populations. Similarly, separate measures might 
be created for general education and major field courses, for courses in particular disciplinary areas, or by year 
of study. An alternative to a single item here is a series of items in which students are asked to report their 
time spent engaging in particular activities. 

ASSOCIATED “GOOD PRACTICE”: Collaborative Learning 

ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is appropriate items on a graduating student survey. A suggested question text is, “In your classes 
as an undergraduate, about how often did you have conversations with other students regarding topics that 
came up in your classes?” 



REFERENCE NUMBER: 8.D.C.2 
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INDICATOR: Percentage of courses with a clear attendance policy. 



DESCRIPTION: This is intended as an indicator of the degree to which requirements established by faculty 
encourage student time on task. Because the source of the indicator is published syllabus material, the statistic 
may not directly reflect the actual presence of such a policy for two reasons: it may be present and not 
documented in the syllabus, or they may be claimed in the syllabus and not actually practiced. Because of 
these drawbacks, the statistic should be interpreted with caution, and student self-reports might be used in 
addition. 



ASSOCIATED ‘‘GOOD PRACTICE”: Time on Task 

ASSOCIATED PERSPECTIVE: Institutional Requirements 

Instructional Delivery 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is collected syllabus material which must be coded manually to reflect the presence of a clear 
attendance policy. Calculation of the statistic then is based on a simple ratio between courses containing such 
policies and the total number of courses. Because of the difficulty of accomplishing such coding, the indicator 
might be calculated only on the basis of a random sample of classes, or for classes that meet particular criteria 
(e.g., 100-level, offered for general education credit, etc.). If a student self-report is used in addition, the 
recommended data source is additional items included in an end-of-course evaluation survey administered to 
all enrolled students. A suggested question text is, “Did the instructor require you to attend classes regularly 
and was attendance counted in your grade for this course?” 



REFERENCE NUMBER: 9.B.a.l 
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INDICATOR: Average student course-load enrolled for. 



DESCRIPTION: This is intended as a rough measure of student commitment to completing a curriculum and 
investing time in education. It is important to emphasize that differences in course load may occur for many 
reasons, not all of them under the control of the student. Nevertheless, considerable research demonstrates that 
students enrolling for what approximates a full load generally experience quite different levels of persistence 
and kinds of experiences from those who do not. As a result, the indicator might be constructed with equal 
utility in terms of the proportions of students in particular credit-load ranges (e.g., three or fewer, three to nine 
credits, etc.) or simply in terms of a full-time/part-time breakdown. 



ASSOCIATED “GOOD PRACTICE”: Time on Task 



ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is the institution’s registration records system for a particular term or academic year. The 
calculational procedure for this statistic is routine, and requires determination of the enrolled credit load at 
census date for each enrolled student during the period in question, averaged over the number of students 
enrolled for at least one credit. 



REFERENCE NUMBER: 9.D.b. 1 
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INDICATOR: Percentage of students reporting that they cut two or fewer meetings of a given class in the 
course of a term. 



DESCRIPTION: This is intended as a direct measure of student commitment to a particular class. As a 
self-report, it is likely to be under-reported rather than exaggerated but is probably reliable as an indicator of 
relative student commitment. If actual attendance records are available, they might be used as a cross-check 
to self-reported data. An alternative might be to construct the indicator not in terms of the absolute number of 
class meetings, but rather in terms of the proportion of total class time that missed classes represent. 



ASSOCIATED “GOOD PRACTICE”: Time on Task 



ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is an additional item to be included in an end-of-course evaluation survey administered to all 
students enrolled in the course. A suggested question text is, “About how many times did you cut or not attend 
this class during the term?” 



REFERENCE NUMBER: 9.D.C. 1 
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INDICATOR: Amount of time in a given class spent studying or working on class assignments. 

DESCRIPTION: This is intended as a direct measure of student time on task associated with coursework. As 
a self-report, it is likely to be an overestimate of actual time invested, but experience suggests that it varies 
reliably with actual student investments. As a result, it is probably the single most useful statistic associated 
with this element of good practice. Class-by-class collection of such data might be usefully supplemented (or 
substituted, if resources are constrained) by items included in surveys administered to graduating seniors. 

ASSOCIATED “GOOD PRACTICE”: Time on Task 

Ongoing Practice 

ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is an additional item included on an end-of-course evaluation survey administered to all enrolled 
students. A suggested question text is, “Excluding classroom hours, about how much time did you spend per 
week doing reading and completing assignments for this class?” For graduating students, a suggested question 
text is, “As an undergraduate, about how many hours per week on average did you spend on assigned reading, 
homework, and the preparation of assignments?” 

REFERENCE NUMBER: 9.D.C.2 
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INDICATOR: Percentage of available library study spaces occupied by students in the period 5:00-9:00 P.M. 



DESCRIPTION: This is intended as an unobtrusive indicator of student out-of-class study activities that can 
be easily collected on a sample basis. As an unobtrusive method, it is subject to considerable limitations of 
interpretation: students may in fact be engaged in considerable study work in residence halls and at home that 
would not be revealed by such inspection. Interpretations may also be asymmetric: empty study spaces does 
not necessarily mean low time on task, but full ones (in relation to enrollment size, of course) probably suggest 
high levels of student investment. 

ASSOCIATED “GOOD PRACTICE” : Time on Task 



ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is periodic direct inspection of the library and/or other public study spaces for students. The 
statistic itself is constructed as a simple ratio between the number of students working in such spaces and the 
total number of such spaces. Alternatively, the denominator might reflect total student enrollment. Inspection 
of this kind should occur on a rotating sample basis, ensuring that different evenings of the week are monitored, 
and perhaps covering different points in the term. 



REFERENCE NUMBER: 9.D.d. 1 
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INDICATOR: Percentage distribution of undergraduate students by race and gender. 

DESCRIPTION: This is intended as an overall contextual measure of opportunities for students to experience 
diverse perspectives through peer contact. Although suggestive as a general benchmark, to be truly useful it 
should be compiled on a program or discipline basis. 

ASSOCIATED “GOOD PRACTICE”: Respect Diversity 

ASSOCIATED PERSPECTIVE: Resource Investments 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is the institution’s registration records system for a given term or academic year. The statistic is 
a straightforward calculation of the proportion of students falling within particular categories on a headcount 
basis. 



REFERENCE NUMBER: lO.A.b.l 
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INDICATOR: Percentage of courses requiring students to experience different learning styles such as 
listening, viewing visual material, etc. 



DESCRIPTION: This is intended as a measure of the degree to which faculty emphasize student interaction 
with course content in a number of different learning modalities. At minimum, this might include a requirement 
to examine visual material, quantitative data, and oral presentations. Separate measures of the occurrence of 
each of these modalities might be examined, or a composite indicator developed that reflects the presence of 
several such modalities. Like similar indicators based on documentary evidence, error can occur in two ways. 
Practices reported in course descriptions may not in fact be present in course delivery, and the requisite 
behaviors may occur without being documented by faculty in the form of syllabus entries. As a result, such 
indicators should always be cross-validated by additional sources of evidence such as those provided by student 
self-reports. 

ASSOCIATED “GOOD PRACTICE”: Respect for Diversity 

ASSOCIATED PERSPECTIVE: Instructional Delivery 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is an analysis of course syllabi for a range of courses of interest. Within each course section, 
assignments and presentations can be coded on the basis of types of learning experiences consistent with 
different learning styles, with the results compiled by type across course sections. Student self-reports and 
faculty surveys might also be used to construct similar indicators. 

REFERENCE NUMBER: 1 0.C.a. 1 
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INDICATOR: Probability that a given student will enroll in at least one class containing more than 20% of 
students drawn from a different ethnic group or background in his or her freshman year. 



DESCRIPTION: This is intended as a more focused measure of the opportunity for contact among students 
of diverse backgrounds provided by the behavioral curriculum. As an overall measure, it can say little about 
the degree to which the implied opportunities of learning from diversity are either experienced or effectively 
harnessed. Nevertheless, it serves as an important contextual indicator for this arena of good practice because 
absent sufficient opportunities for contact, little else can occur. 



ASSOCIATED “GOOD PRACTICE” : Respect for Diversity 

Early Years of Study 



ASSOCIATED PERSPECTIVE: Instructional Delivery 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this measure is the institution’s registration records system, though considerable manipulation of base data will 
be required. Generating the statistic requires first that individual student demographic information can be 
linked to particular course enrollment records. If this match can in fact be accomplished, the calculation 
requires, a) determination of which set of classes each student in the target group (e.g., first year students) 
enrolled for, b) identification of the particular students enrolled in each course so identified, c) calculation of 
a demographic profile (excluding the student in question) for each course that the student enrolled for, d) 
application of a criterion of 20% of students other than the target student’s own ethnicity, and e) determination 
of whether any classes in the student’s record for the identified time period meet the criterion noted in d. The 
resulting statistic is the ratio between those students meeting the criterion and the total target student population. 
Alternatively, a student self-report might be used, either on an end-of-courses evaluation survey administered 
to enrolled students or on a graduating student survey. 



REFERENCE NUMBER: lO.C.b. 1 
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INDICATOR: Percentage of faculty reporting that they regularly use individualized or alternative forms of 
instruction to communicate course materials. 



DESCRIPTION: The intent of this indicator is to reflect the overall incidence of faculty practices designed 
explicitly to communicate effectively with students drawn from different backgrounds or who learn in different 
ways. Examples of this practice are many, but might include the provision of alternate ways for students to 
demonstrate mastery of course materials such as examinations or hands-on projects and oral presentations, the 
provision that each student be required to demonstrate mastery in several ways, or a commitment to use 
alternative presentation or pedagogical formats suited to a range of different learning styles. As an instructor 
self-report, it is subject to the usual limitations. Faculty may report such practices but in fact not engage in 
them. Similarly, they may engage in such practices, but not recognize the fact that they are doing so. As a 
result, data drawn from this source should be cross-validated with additional evidence drawn from student 
self-reports and syllabus review wherever possible. 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is a survey administered periodically to instructional staff. A suggested question text is, “In your 
classes this year, to what degree did you try to vary teaching methods and presentations, or to provide additional 
course material, in order to accommodate the needs of students who learn in different ways?” 



ASSOCIATED “GOOD PRACTICE”: Respect Diversity 



ASSOCIATED PERSPECTIVE: 



Instructional Delivery 



REFERENCE NUMBER: 



lO.C.c.l 
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INDICATOR: Percentage of students reporting that they were encouraged to ask questions in class when they 
did not understand something. 



DESCRIPTION: This is intended as a measure of the degree to which faculty create a classroom atmosphere 
that legitimizes questions arising from different points of view. As a self-reported measure, it does not capture 
the degree to which faculty act on such information but it represents a good overall indicator of faculty 
willingness to seek feedback from students about whether presented material is being effectively communicated 
to students who may come from a variety of backgrounds and learning styles. Consistent with this, a parallel 
indicator might be faculty practice of explicit “classroom research” techniques designed to tap diverse student 
perceptions and approaches to learning (Angelo and Cross 1993). 



ASSOCIATED ‘‘GOOD PRACTICE” : Respect Diversity 

Active Learning 



ASSOCIATED PERSPECTIVE: Student Experience 

[Instructional Delivery] 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE. The data source for 
this indicator is an additional item on an end-of-course evaluation survey administered to all enrolled students. 
A suggested question text is, “Did the instructor in this class encourage you to ask questions in or after class 
when there was something that you did not understand.” 

REFERENCE NUMBER: lO.D.c.l 
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INDICATOR: Percentage of students reporting that the grading and evaluation procedures used by the 
instructor allowed them to actually demonstrate what they knew. 



DESCRIPTION: This is intended as a measure of student perception of the degree to which the instructor’s 
(and/or the institution’s) assessment practices reflect diverse ways of demonstrating mastery. Asa student 
self-report, it is subject to considerable limitations because students may attribute unfavorable instructor 
assessments to simple lack of understanding of their own points of view. Nevertheless, as a contextual measure, 
this indicator is useful as a benchmark of the degree to which faculty and student perceptions of achievement 
coincide. Where they do not, additional investigation as to whether faculty have in fact examined diverse ways 
of demonstrating mastery of course material may be warranted. 



ASSOCIATED ‘‘GOOD PRACTICE”: Respect Diversity 

ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator as an additional item on an end-of-course evaluation survey administered to all students enrolled. 
A suggested question text is, “Did the grading and evaluation procedures used in this course allow you to fairly 
demonstrate what you learned or could do related to the topics covered?” 



REFERENCE NUMBER: lO.D.c.2 
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INDICATOR: Percentage of graduating students reporting that their undergraduate experiences reflected a 
concern and appreciation for diverse backgrounds and perspectives. 



DESCRIPTION: This is intended as an overall measure of student perceptions of the degree to which the 
institution encourages a respect for alternative viewpoints. 



ASSOCIATED “GOOD PRACTICE”: Respect for Diversity 

ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this procedure is an item or items included on a graduating student survey. A suggested question text is, “In 
the courses you took as an undergraduate, did your instructors use varied methods to get the material across, 
or did they all teach in pretty much the same way?” 



REFERENCE NUMBER: lO.D.c.3 
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INDICATOR: Faculty advising load [by program or department]. 



DESCRIPTION: This is intended as a direct measure of a standard faculty opportunity to interact with 
students. Large numbers of advisees per instructor will tend to degrade the quality of the advising encounter, 
absent other factors. Because this is a statistic derived directly from institutional records, however, there is no 
way to determine the quality of advising in situations where the numbers could allow it to occur. Benchmarks 
for this indicator are best set internally, based on institutional experience. Where advising loads exceed thirty 
students per faculty member, however, student/faculty encounters of this kind will probably begin to suffer. 



ASSOCIATED “GOOD PRACTICE”: Student/Faculty Contact 



ASSOCIATED PERSPECTIVE: Resource Investments 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is the institution’s student records system — provided that this system contains data elements that 
identify each student’s advisor. Absent such data elements, compiling the indicator will probably require 
accessing the records of individual departments or academic programs. The construction of the indicator is 
straightforward and consists of a simple ratio between the number of instructional staff engaged in advising 
students and the number of advisees. Adjustments to the calculation might use FTE equivalents for counting 
faculty. Also, the statistic is probably most useful when broken down by program or department. Alternatively, 
an item on a faculty survey might be constructed. 



REFERENCE NUMBER: 1 1 . A.b. 1 
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INDICATOR: Average number of undergraduate students enrolled in a program or major per full-time 
equivalent faculty assigned to that program or major. 



DESCRIPTION: This is a general measure of the overall potential for meaningful student/faculty contact 
within a given program or department. Because it says nothing about actual faculty deployment or student 
behavior, it should be considered nothing more than an indicator of the potential for such contact. It is useful 
at the major field level principally as an indicator of advising load. If the ratio is very high, the likelihood of 
high levels of contact is low no matter what the institution’s policies may be. 



ASSOCIATED “GOOD PRACTICE” : Student/Faculty Contact 

ASSOCIATED PERSPECTIVE: Institutional Requirements 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is regular institutional registration records for a given term or academic year. The calculational 
procedure used is conceptually straightforward, but may allow many variations. The most common is to count 
students majoring in the program on a headcount basis, and numbers of available full-time faculty on also on 
a headcount basis. An alternative for the latter is to only include those faculty who have an assignment that 
affects undergraduate students (teaching or advising), or to construct an FTE measure based upon actual 
assignments to these functions. If the latter is used, part-time faculty members might also be included in the 
ratio. 
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INDICATOR: Average section size (excluding independent study sections) by program or level. 



DESCRIPTION: This indicator is intended as a very general measure of the potential for student/faculty 
contact occurring, Because it does not reflect actual faculty or student behavior, it is highly indirect. Also, as 
a central tendency measure, it is unable to reflect the fact that some very small sections may in fact be available 
to students, masked in the overall average. Against this, it is a very straightforward statistic to calculate, and 
can easily be obtained from existing institutional records. This indicator is most useful when it is used to 
describe a particular subset of sections offered rather than being calculated for the institution as a whole. For 
example, it might be beneficially constructed for individual departments or programs, for broad categories of 
courses (e.g., those counting for general education credit in general or in a particular distribution area), or for 
level of offering (e.g., first-year 100-level, intermediate 200/300-level, etc.). 



ASSOCIATED “GOOD PRACTICE” : Student/Faculty Contact 

Collaborative Learning 
Assessment and Feedback 



ASSOCIATED PERSPECTIVE: Instructional Delivery 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is regular institutional registration records for a given term or academic year. The calculational 
procedure used is straightforward, and involves determining the total number of course registrations occurring 
within the domain of interest (e.g., a particular department’s courses, all 100-level courses, all courses counting 
for “social science distribution” within the institution’s general education requirements, etc.) and dividing this 
number by the total number of sections offered within the same domain. Any sections that are clearly intended 
as independent study or that involve other one-on-one contact (e.g., music lessons, required practice interviews, 
etc.) are typically excluded from the calculation — ^though it may be of interest to construct an indicator that 
reflects only these types of faculty/student interactions. This requires such sections to be identifiable by code 
or number within the institution’s registration or student records system. 
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INDICATOR: Percentage of sections offered with fewer than X number of student enrollments [by program 
or level]. 



DESCRIPTION: This indicator is intended as a more precise measure of the potential for student/faculty 
contact than average section size. Like average section size, however, it is only a measure of such possible 
contact and does not reflect either faculty or student behavior. Like average section size, it is most useful when 
it is calculated for a particular subset of courses — for example those offered by a particular department, required 
for a particular program, reflecting a given requirement (e.g., an identified area of general education 
distribution), or offered at a particular level. The number set as a criterion can vary, but given the intent to 
indicate favorable opportunities for student/faculty contact, should not exceed twenty students and should most 
thiitfully be set at fifteen students. 



ASSOCIATED “GOOD PRACTICE”: Student/Faculty Contact 

ASSOCIATED PERSPECTIVE: Instructional Delivery 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is regular student registration records for a given term or academic year. The calculational 
procedure used is similar to that employed for average section size and involves, a) determining the total number 
of course sections within the domain of interest, b) determining the number of these with enrollments of less 
than X students and, c) dividing the number meeting the enrollment criterion by the total number within the 
domain of interest. As in the calculation of average section size, it is best to exclude “independent study” and 
similar sections fi'om this calculation and if this is to be accomplished, a code or section identifier is required 
to flag these courses. 
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INDICATOR: Percent of faculty knowing at least half of the students in their classes by name by the end of 
X weeks into the term. 



DESCRIPTION: This is intended as a measure of the effort faculty have invested in becoming personally 
knowledgeable about their students. Because it is a self-report, it should be treated with some caution and it 
does not reflect the frequency or quality of student/faculty contact. Nevertheless, it is a reasonable indicator 
of at least a minimum level of faculty investment. Faculty reports can be taken at any time during the term so 
long as the snapshot is on a consistent basis, but will be most indicative within the first third of the 
term — probably within the first two weeks. An analogous indicator might be constructed on the basis of student 
self-reports taken at the end of the term. 

ASSOCIATED “GOOD PRACTICE”: Student/Faculty Contact 

ASSOCIATED PERSPECTIVE: Instructional Delivery 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE. The data source for 
this indicator is a periodic survey of instructional staff. A suggested question text is “About what proportion 
of the students whom you are teaching this year would you recognize and know by name outside of class?” 
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INDICATOR: Percent of faculty reporting involvement with a student club or organization. 

DESCRIPTION: This is intended as a self-reported measure of faculty engagement with students outside the 
classroom. Because it is a self-report of an activity for which faculty may or may not keep records, it is an 
approximation only. Nor does it directly reflect the degree of actual contact or mentoring which may take place 
in these settings. 



ASSOCIATED “GOOD PRACTICE”: Student/Faculty Contact 

ASSOCIATED PERSPECTIVE: Instructional Delivery 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is a periodic survey of instructional staff A suggested question text is, “Did you participate 
formally or informally in any student clubs, organizations, or other groups this year?” 
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INDICATOR: Average number of hours/week spent advising or meeting with students on academic matters. 

DESCRIPTION: This is intended to reflect the level of faculty effort expended on out-of-class contact with 
students. It is a self-reported measure, and therefore does not reflect either “customer” reactions and 
satisfactions, or the ultimate quality of the encounter. It should also be used in conjunction with advisee load 
to determine if high values for this indicator are associated with large numbers of students to be served, or 
rather with significant contact with individual students. 



ASSOCIATED “GOOD PRACTICE” : Student/Faculty Contact 

ASSOCIATED PERSPECTIVE: Instructional Delivery 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is a periodic survey administered to instructional staff A suggested question text is, “In addition 
to classroom teaching, preparation, and grading assignments, about how many hours per week did you spend 
advising or meeting with students on matters related to their academic work this year?” 
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INDICATOR: Percent of students reporting after-class conversations on academic subjects with faculty 
members. 



DESCRIPTION: This is intended as a relatively direct measure of actual student/faculty interaction in the 
most frequent setting for such encounters. Reported on a class-by-class basis, it is probably the single best 
measure available for assessing such contact. 



ASSOCIATED “GOOD PRACTICE”: Student/Faculty Contact 



ASSOCIATED PERSPECTIVE: Student Experiences 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is as an additional item on an end-of-course evaluation survey administered to all students enrolled 
in the class. A suggested question text is, “About how many times during the term did you stay after class to 
make a comment or to ask the instructor a question?” 
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INDICATOR; Percentage of students reporting having visited faculty during established office hours. 

DESCRIPTION: This is also intended as a direct measure of the frequency of student/faculty contact in a 
common setting for such contact. It is a self-reported indicator intended to be collected on a class-by-class 
basis. 

ASSOCIATED “GOOD PRACTICE”: Student/Faculty Contact 

ASSOCIATED PERSPECTIVE: Student Experiences 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is as an additional item on an end-of-course evaluation survey administered to all students enrolled 
in the class. A suggested question text is, “About how many times during the term did you see your instructor 
in his or her office to discuss a topic related to this course?” 
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INDICATOR: Average rated quality of out-of-class contact with faculty. 



DESCRIPTION: This is intended as a “customer rating” of the quality of student/faculty interaction collected 
directly from students on a class-by-class basis. While a measure of student satisfaction, it is not necessarily 
a measure of the educational efficacy or actual impact of any such contact. 



ASSOCIATED “GOOD PRACTICE”: Student/Faculty Contact 



ASSOCIATED PERSPECTIVE: Student Experiences 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is an additional item on an end-of-course evaluation survey administered to all students enrolled 
in the course. Rating on a five-point scale with an additional “not applicable” category is recommended. A 
suggested question text is, “How helpful did you find any out-of-class discussions that you may have had with 
the instructor about this course?” 
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INDICATOR: Overall frequency [and quality] of out-of-class contact with faculty in the total undergraduate 
experience. 



DESCRIPTION: This is intended as a summary retrospective measure of student/faculty contact collected 
from graduating seniors. Like other self-reported measures of this kind, it constitutes a good indicator of 
“customer satisfaction” but may be less reliable as a measure of the efficacy of such contact. Because it is a 
recollected item, it is less reliable than collecting similar data on a class-by-class basis but is easier to obtain. 
A useful variant might be to construct separate sets of items for general education and the major field, or for 
selected departments. 



ASSOCIATED “GOOD PRACTICE”: Student/Faculty Contact 

ASSOCIATED PERSPECTIVE: Student Experiences 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is through items included on a graduating student survey. A suggested question text for assessing 
frequency is, “In addition to your classes, how often did you generally have a conversation with a faculty 
member on an academic topic.” For rating quality, a five-point scale should be used together with a “not 
applicable” category, and a suggested question text, “Throughout your undergraduate experience, how would 
you describe the quality of your interaction with faculty members outside the classroom?” 
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INDICATOR: Ratio of instructional expenditures per FTE student associated with 100-leve! [or 
lower-division] courses with that for all courses. 



DESCRIPTION: This is intended as an indirect measure of the level of resource investment made by an 
institution or program in the early years of study. Because it is a measure of investment only, it does not address 
the degree to which effective educational practices were actually used or if student experiences were in fact 
different as a result of such expenditures. Nevertheless, it remains a straightforward and highly credible 
indicator of institutional or program commitment. 



ASSOCIATED “GOOD PRACTICE”: Early Years of Study 



ASSOCIATED PERSPECTIVE: Resource Investments 



RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is the institution’s registration records system together with expenditure records in the form of 
an instructional cost study. Absent an existing cost study rubric, construction of the measure is complex and 
requires, a) identification of the costs associated with each course section through individual faculty assignment 
and the proportion of salary, b) identification of any additional costs associated with the class, and c) 
determination of the number of credits produced by the section. Calculation of the statistic requires dividing 
the cost by the total FTE associated with the course section for 100-level [or lower-division] courses and 
comparing the result with a parallel statistic constructed for all courses. 
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INDICATOR: Proportion of 100-level [or lower-division] courses taught by full-time, tenure-track faculty. 



DESCRIPTION: This is intended as a direct measure of the institution’s relative commitment of instructional 
resources to the early years of study. As such, it is highly credible to external audiences despite the fact that 
it may or may not reflect heightened effectiveness absent additional investments in pedagogy on the part of 
regular faculty. Nevertheless, because it is easy to collect and directly references institutional commitment, it 
remains probably one of the most effective single indicators associated with this “good practice” domain. 



ASSOCIATED “GOOD PRACTICE” : Early Years of Study 

ASSOCIATED PERSPECTIVE: Instructional Delivery 

Use of Resources 



RECOMMENDED DATA SOURCE AND/OR CALCULATION AL PROCEDURE: The data source for 
this indicator is the institution’s registration or student records system for a given term or academic year. 
Construction of the indicator requires, a) identification of all course sections delivered at the 100-level [or 
lower-division level], b) determination of the instructor associated with each section, and c) classification of 
the instructor as appropriate. The resulting statistic is the proportion of classes within the identified domain 
taught by a regular faculty member. Accomplishing this calculation will require the ability to cross-walk course 
records with faculty data. 
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INDICATOR: Probability of first-time freshman enrolling in at least one course section with X or fewer 
students. 



DESCRIPTION : This indicator is intended as a direct measure of the effectiveness of an institution’s emphasis 
on the early years of study. At the same time, it is a further measure of the potential for student/faculty contact 
within the curriculum. The emphasis of the indicator is on measuring real behavioral opportunity rather than 
simply the deployment of resources. As such, it might be constructed for any student subpopulation including 
demographic groups (e.g., minority students, students deficient in writing skills) or groups based on the 
student’s interaction with the institution (e.g., transfer students, upper-division students, or students enrolled 
in particular majors or programs). Given a sufficient database, it might also record the number of instances of 
such small-class enrollments over an entire undergraduate career. As in previous examples, the measure would 
typically exclude independent study sections and the criterion might be set at any level. Constructed as shown, 
it is also an indicator of investments consistent with an additional good practice — ^the potential for collaborative 
learning. 



ASSOCIATED “GOOD PRACTICE”: Early Years of Study 

Student/Faculty Contact 
Collaborative Learning 



ASSOCIATED PERSPECTIVE: Instructional Delivery 

RECOMMENDED DATA SOURCE AND/OR CALCULATIONAL PROCEDURE: The data source for 
this indicator is existing institutional registration or student records for a given set of terms or academic years. 
Constructing the measure is complex, however, and requires, a) identifying the student population of interest 
(for example, an entering cohort of first-time freshmen), b) determining the specific course sections enrolled 
in by each member of the identified population during the specified time period through transcript records, c) 
determining the total class size for each course section so identified, and d) flagging students who meet the set 
criterion of enrolling for at least one course at or below the established limit, and e) calculating the required 
probability by dividing the number of students meeting the criterion by the total number in the subpopulation. 
Accomplishing this calculation will require the ability to cross-walk course enrollment and student 
demographic data files — ^a capability not always present. 
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INDICATOR: Percentage of first-year students reporting participation in a comprehensive and helpful 
orientation or advising program. 



DESCRIPTION: This is intended as a measure of the actual coverage and participation in any programs 
designed explicitly to enhance the success of first-year students. Examples of relevant activities under this 
heading might include formal “freshman year experience” programs, pre-enrollment orientation programs of 
significance (e.g., more than two days in duration and/or with multiple occurrences across the first year), 
advising sessions designed explicitly to ease the transition to collegiate life, etc. The indicator is based on 
student self-report, but an alternative approach is to actually examine the presence of and participation in such 
activities through institutional records. Given that a self-report is used, moreover, items might be constructed 
that reflect not only participation in such activities but also the frequency with which ongoing advisement 
occurred throughout the first year, and/or student ratings of its quality or helpfulness. 

ASSOCIATED “GOOD PRACTICE”: Early Years of Study 

ASSOCIATED PERSPECTIVE: Student Experience 

RECOMMENDED DATA SOURCE AND/OR C ALCULATIONAL PROCEDURE: The data source for 
this indicator is a questionnaire administered to first-year students. A suggested question text is, “What specific 
activities were available to you, in which you participated, that were designed to introduce you to the college 
and get you started successfully with your academic work this year?.. .How helpful did you find participation 
in each of these activities?” 
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What NCHEMS Offers You 

Established to meet the needs of working administrators, NCHEMS delivers research-based expertise, 
practical experience, information, and a range of management tools that can help institutions improve both 
their efficiency and their effectiveness. NCHEMS places these resources in the hands of administrators 
through a variety of means: 

. Through specific research, consulting, or development projects funded by institutions, consortia, state 
agencies, federal contracts, or foundations 

. Through a variety of information services that make the Center’s extensive data holdings accessible to 
the higher education community 

. Through numerous publications that disseminate concepts, principles, and strategies to a wide audience 
of administrators and researchers. 

Through an extensive membership program, NCHEMS helps foster two-way communication with its 
constituents, and provide an opportunity for the Center to understand and meet the changing needs of 
postsecondary institutions. Responsiveness to higher education’s practical needs is also a natural product of 
what we do: the greatest strength of NCHEMS remains its constant working contact with higher education 
practitioners in the direct solution of common problems. 

NCHEMS subsidiary, NCHEMS Management Services, Inc. (NMSl), offers individual consulting 
services and an integrated series of seminars designed to help college and university administrators address 
a range of typical management problems. 



SERVICES 

General Information - (303) 497-0301 

Information Services 

Orders and Inquiries - (303) 497-03 14 

Management Consulting - (303) 497-0301 

Management Seminars 

Information and Registration - (303) 497-0345 

Membership 

Information and Enrollment - (303) 497-0321 

Publications 

Orders and Information - (303) 497-0390 

FAX (NCHEMS & NMSI) - (303) 497-0338 
EMAIL: nchems@spot.colorado.edu 
WEB: http://www.nchems.com 




National Center for Higher Education Management Systems 
P.O. Box 9752 Boulder, CO 80301-9752 
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